Skip to content

Commit 5b17740

Browse files
syntheaddylan-smith
synthead
and
dylan-smith
authored
Add --download-migration-logs option to generate-script command for gei and ado2gh (#382)
* Add --download-migration-logs option to generate-script. * Wrap a couple commands in Exec in GenerateScriptCommand. * Remove unused param to MigrateGithubRepoScript in GenerateScriptCommand. * Add more tests for GenerateScriptCommand with migration logs. * Remove extra param from MigrateGithubRepoScript calls. * Add release notes for --download-migration-logs option. * Add --download-migration-logs to ado2gh generate-script. * Always check exit code of migrate-repo in parallel ADO scripts. * Add tests for downloading migration logs to GenerateScriptCommandTests. * Pass --download-migration-logs to integration tests. * Fix possibly null variable in GenerateScriptCommandTests. * Add DownloadMigrationLogs to GenerateScriptOptions in GenerateScriptCommand. * Remove some unused parameters from GenerateScriptCommand. * Include download-logs command ado2gh generate-script's --all. * Add AssertMigrationLogFileExists integration test helper. * Add ado2gh integration tests to ensure migration logs exist. * Add gei integration tests to ensure migration logs exist. * Use better log message for AssertMigrationLogFileExists helper. * Use async Task in AssertMigrationLogFileExists helper. * Add --wait option to download-logs command for gei. * Add --wait option to download-logs command for ado2gh. * Add --wait to download-logs in gei GenerateScriptCommand. * Add --wait to download-logs in ado2gh GenerateScriptCommand. * Remove unnessary assignments from DownloadLogsCommand classes. * Fix style to ifs in DownloadLogsCommand classes. * Un-hide download-logs commands from gei and ado2gh. * Update gei DownloadLogsCommand to always retry with a timeout. * Update ado2gh DownloadLogsCommand to always retry with a timeout. * Add comment to timeoutMinutes var in DownloadLogsCommandTests. * Check for --timeout-minutes param in DownloadLogsCommandTests. * Don't pass --wait to download-logs in GenerateScriptCommand. * Don't pass --wait to download-logs in ado2gh GenerateScriptCommand. * Use Pascal case in names for all tests. * Use var instead of explicit type in DownloadLogsCommand classes. * Add readonly to DateTimeNow Funcs. * Fix Pascal case on a couple DownloadLogsCommand tests. * Use hard-coded timeout in gei DownloadLogsCommand. * Remove unused variable in gei DownloadLogsCommandTests. * Use hard-coded timeout in ado2gh DownloadLogsCommand. * Use constants in DownloadLogsCommandTests test classes. * Add newline in a test for gei DownloadLogsCommand. * Update grammar in DownloadLogsCommand classes. * Use syntactic sugar null check in DownloadLogsCommand classes. * Remove async from AssertMigrationLogFileExists. * Use BeTrue() in AssertMigrationLogFileExists. * Use implicit type in for loops in DownloadLogsCommand classes. * Remove redundant cast to string in DownloadLogsCommandTests classes. * Add 24h availability warning to gei DownloadLogsCommand. * Add 24h availability warning to ado2gh DownloadLogsCommand. * Remove null params to Invoke in DownloadLogsCommandTests classes. * Remove --download-migration-logs from ado2gh integration tests. * Don't include org name in start of migration log output. * Don't test for log output in DownloadLogsCommand classes. * Call download-logs command after other ado2gh commands. co-authored-by: Dylan Smith <[email protected]> * Add DOWNLOAD MIGRATION LOGS output in ado2gh generate-script. * adopted Polly for retry logic * Refactor gei test for single ADO repo with migration logs. * Refactor gei test for ADO server with single repo and migration logs. * Refactor remaining gei GenerateScriptCommandTests. * Join GetOsDistPath() in AssertMigrationLogFileExists. * Make _httpRetryInterval readonly in RetryPolicy. * Use Polly in gei DownloadLogsCommand. * Remove unused "using" in DownloadLogsCommandTests. * Fix broken tests from merging main. * Add verbose logs to HttpDownloadService. * Fix curly location in HttpDownloadService. Co-authored-by: Dylan Smith <[email protected]>
1 parent 6c648a6 commit 5b17740

16 files changed

+982
-373
lines changed

‎RELEASENOTES.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1+
- Add `download-logs` command to download migration logs
2+
- Add `--download-migration-logs` option to `generate-script` command
13
- Log GitHub request id into the verbose log for each GitHub API call (this can be useful for GitHub support if something goes wrong)

‎src/Octoshift/GithubApi.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ public virtual async Task<IEnumerable<string>> GetTeamMembers(string org, string
8686
{
8787
var url = $"{_apiUrl}/orgs/{org}/teams/{teamSlug}/members?per_page=100";
8888

89-
return await _retryPolicy.Retry(async () => await _client.GetAllAsync(url).Select(x => (string)x["login"]).ToListAsync(),
89+
return await _retryPolicy.HttpRetry(async () => await _client.GetAllAsync(url).Select(x => (string)x["login"]).ToListAsync(),
9090
ex => ex.StatusCode == HttpStatusCode.NotFound);
9191
}
9292

@@ -290,7 +290,7 @@ public virtual async Task<string> GetMigrationState(string migrationId)
290290

291291
var payload = new { query = $"{query} {{ {gql} }}", variables = new { id = migrationId } };
292292

293-
var response = await _retryPolicy.Retry(async () => await _client.PostAsync(url, payload),
293+
var response = await _retryPolicy.HttpRetry(async () => await _client.PostAsync(url, payload),
294294
ex => ex.StatusCode == HttpStatusCode.BadGateway);
295295
var data = JObject.Parse(response);
296296

@@ -349,7 +349,7 @@ public virtual async Task<string> GetMigrationFailureReason(string migrationId)
349349

350350
var payload = new { query = $"{query} {{ {gql} }}", variables = new { id = migrationId } };
351351

352-
var response = await _retryPolicy.Retry(async () => await _client.PostAsync(url, payload),
352+
var response = await _retryPolicy.HttpRetry(async () => await _client.PostAsync(url, payload),
353353
ex => ex.StatusCode == HttpStatusCode.BadGateway);
354354
var data = JObject.Parse(response);
355355

‎src/Octoshift/HttpDownloadService.cs

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,22 @@ public class HttpDownloadService
1111
{
1212
internal Func<string, string, Task> WriteToFile = async (path, contents) => await File.WriteAllTextAsync(path, contents);
1313

14+
private readonly OctoLogger _log;
1415
private readonly HttpClient _httpClient;
1516

16-
public HttpDownloadService(HttpClient httpClient) => _httpClient = httpClient;
17+
public HttpDownloadService(OctoLogger log, HttpClient httpClient)
18+
{
19+
_log = log;
20+
_httpClient = httpClient;
21+
}
1722

1823
public virtual async Task Download(string url, string file)
1924
{
25+
_log.LogVerbose($"HTTP GET: {url}");
26+
2027
using var response = await _httpClient.GetAsync(url);
28+
_log.LogVerbose($"RESPONSE ({response.StatusCode}): <truncated>");
29+
2130
response.EnsureSuccessStatusCode();
2231

2332
var contents = await response.Content.ReadAsStringAsync();

‎src/Octoshift/RetryPolicy.cs

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,21 +8,34 @@ namespace OctoshiftCLI
88
public class RetryPolicy
99
{
1010
private readonly OctoLogger _log;
11+
internal readonly int _httpRetryInterval = 1000;
12+
internal int _retryOnResultInterval = 4000;
1113

1214
public RetryPolicy(OctoLogger log)
1315
{
1416
_log = log;
1517
}
1618

17-
public async Task<T> Retry<T>(Func<Task<T>> func, Func<HttpRequestException, bool> filter)
19+
public async Task<T> HttpRetry<T>(Func<Task<T>> func, Func<HttpRequestException, bool> filter)
1820
{
1921
var policy = Policy.Handle(filter)
20-
.WaitAndRetryAsync(5, retry => retry * TimeSpan.FromMilliseconds(1000), (ex, _) =>
22+
.WaitAndRetryAsync(5, retry => retry * TimeSpan.FromMilliseconds(_httpRetryInterval), (ex, _) =>
2123
{
2224
_log.LogVerbose($"Call failed with HTTP {((HttpRequestException)ex).StatusCode}, retrying...");
2325
});
2426

2527
return await policy.ExecuteAsync(func);
2628
}
29+
30+
public async Task<PolicyResult<T>> RetryOnResult<T>(Func<Task<T>> func, T resultFilter, string retryLogMessage)
31+
{
32+
var policy = Policy.HandleResult(resultFilter)
33+
.WaitAndRetryAsync(5, retry => retry * TimeSpan.FromMilliseconds(_retryOnResultInterval), (_, _) =>
34+
{
35+
_log.LogVerbose(retryLogMessage ?? "Retrying...");
36+
});
37+
38+
return await policy.ExecuteAndCaptureAsync(func);
39+
}
2740
}
2841
}

‎src/OctoshiftCLI.IntegrationTests/AdoToGithub.cs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,8 @@ public async Task Basic()
9090
await _helper.AssertPipelineRewired(adoOrg, teamProject2, pipeline2, githubOrg, $"{teamProject2}-{teamProject2}");
9191
await _helper.AssertBoardsIntegrationConfigured(adoOrg, teamProject1);
9292
await _helper.AssertBoardsIntegrationConfigured(adoOrg, teamProject2);
93+
_helper.AssertMigrationLogFileExists(githubOrg, $"{teamProject1}-{teamProject1}");
94+
_helper.AssertMigrationLogFileExists(githubOrg, $"{teamProject2}-{teamProject2}");
9395
}
9496

9597
protected virtual void Dispose(bool disposing)

‎src/OctoshiftCLI.IntegrationTests/GithubToGithub.cs

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,15 @@ public async Task Basic()
4747
await _helper.CreateGithubRepo(githubSourceOrg, repo1);
4848
await _helper.CreateGithubRepo(githubSourceOrg, repo2);
4949

50-
await _helper.RunGeiCliMigration($"generate-script --github-source-org {githubSourceOrg} --github-target-org {githubTargetOrg}");
50+
await _helper.RunGeiCliMigration($"generate-script --github-source-org {githubSourceOrg} --github-target-org {githubTargetOrg} --download-migration-logs");
5151

5252
await _helper.AssertGithubRepoExists(githubTargetOrg, repo1);
5353
await _helper.AssertGithubRepoExists(githubTargetOrg, repo2);
5454
await _helper.AssertGithubRepoInitialized(githubTargetOrg, repo1);
5555
await _helper.AssertGithubRepoInitialized(githubTargetOrg, repo2);
56+
57+
_helper.AssertMigrationLogFileExists(githubTargetOrg, repo1);
58+
_helper.AssertMigrationLogFileExists(githubTargetOrg, repo2);
5659
}
5760

5861
protected virtual void Dispose(bool disposing)

‎src/OctoshiftCLI.IntegrationTests/TestHelper.cs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -585,5 +585,14 @@ public async Task AssertBoardsIntegrationConfigured(string adoOrg, string teamPr
585585
boardsConnection.Should().NotBeNull();
586586
boardsConnection.repoIds.Count().Should().Be(1);
587587
}
588+
589+
public void AssertMigrationLogFileExists(string githubOrg, string repo)
590+
{
591+
_output.WriteLine("Checking that the migration log was downloaded...");
592+
593+
var migrationLogFile = Path.Join(GetOsDistPath(), $"migration-log-{githubOrg}-{repo}.log");
594+
595+
File.Exists(migrationLogFile).Should().BeTrue();
596+
}
588597
}
589598
}

‎src/OctoshiftCLI.Tests/HttpDownloadServiceTests.cs

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ namespace OctoshiftCLI.Tests
1111
{
1212
public class HttpDownloadServiceTests
1313
{
14+
private readonly Mock<OctoLogger> _mockOctoLogger = TestHelpers.CreateMock<OctoLogger>();
15+
1416
[Fact]
1517
public async Task Downloads_File()
1618
{
@@ -38,7 +40,7 @@ public async Task Downloads_File()
3840
using var httpClient = new HttpClient(mockHttpHandler.Object);
3941

4042
// Act
41-
var httpDownloadService = new HttpDownloadService(httpClient)
43+
var httpDownloadService = new HttpDownloadService(_mockOctoLogger.Object, httpClient)
4244
{
4345
WriteToFile = (_, contents) =>
4446
{
@@ -76,7 +78,7 @@ public async Task Raises_Exception_When_File_Cannot_Be_Downloaded()
7678
using var httpClient = new HttpClient(mockHttpHandler.Object);
7779

7880
// Act
79-
var httpDownloadService = new HttpDownloadService(httpClient)
81+
var httpDownloadService = new HttpDownloadService(_mockOctoLogger.Object, httpClient)
8082
{
8183
WriteToFile = (_, contents) =>
8284
{

0 commit comments

Comments
 (0)