Skip to content

Conversation

jnm2
Copy link
Contributor

@jnm2 jnm2 commented May 31, 2025

dotnet test tools/ExampleTester now runs and produces the same output as dotnet run tools/ExampleTester (which still works), except that it will parallelize the tests and finish in a fraction of the time that Program.Main takes.

You can also click and run a given example in the UI of any .NET test runner, and see red and green states, and see output per test, add a few examples to a test session/playlist for repeated runs, etc. ...these are just standard unit tests.

I've noticed that the GitHub checks are all pretty fast except this one which takes about 9 minutes. We may want to consider switching over the GitHub Actions job to invoke the tests the new way. Even if the build server doesn't parallelize particularly well, simply overlapping the various stages of each test should produce benefits. (Loading csproj, getting compilation, getting diagnostics, emitting, running output.)

Comment on lines +15 to +16
<PackageReference Include="NUnit" Version="4.3.2" />
<PackageReference Include="NUnit3TestAdapter" Version="5.0.0" />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not figure out a way to get xUnit to parallelize between individual cases within a theory. Also, the Visual Studio Test Explorer UI for an xUnit theory does not show theory cases as individual nested tests, so they can't be individually run or added to playlists. NUnit shines in this area.

@jnm2
Copy link
Contributor Author

jnm2 commented May 31, 2025

💭 We could consider having these tests read directly from the standard without going through the ExampleExtractor step. That could confer a few benefits such as possibly removing the use of the very slow MSBuildWorkspace, keeping cached Roslyn compilations around in memory that match each project template. This could result in very fast tests. The downside would be that you wouldn't be able to open the .csproj files in an IDE to play around with things, if there is no tmp folder.

Edit: I was able to get the very fast runs without disrupting the flow via .csproj files on disk: #1338

…noring 'AutoGeneratedProgram.Main(string[])' entry point."
@jnm2 jnm2 marked this pull request as draft June 1, 2025 13:31
@jnm2 jnm2 force-pushed the jnm2/tools2 branch 4 times, most recently from 59e9ce1 to a0e1166 Compare June 1, 2025 13:39
@BillWagner BillWagner self-requested a review June 11, 2025 20:53
Copy link
Member

@BillWagner BillWagner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM.

I'll also ping @jskeet for review offline. I think we can merge and update this without spending committee time.

@BillWagner BillWagner requested a review from jskeet July 29, 2025 19:48
@BillWagner BillWagner closed this Aug 8, 2025
@BillWagner BillWagner reopened this Aug 8, 2025
@jnm2
Copy link
Contributor Author

jnm2 commented Sep 4, 2025

@jskeet Would you be willing to take a look?

@jskeet
Copy link
Contributor

jskeet commented Sep 4, 2025

Yup, I'll look tomorrow.

Copy link
Contributor

@jskeet jskeet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still coldy, but yes, I don't see problems with this so long as we still actually get consistent results :)

(We're generating everything into separate directories, so parallelizing that part should be fine - and if I understand correctly the actual execution is serial, so that shouldn't interfere.)

@jnm2 jnm2 merged commit f075984 into draft-v8 Sep 9, 2025
9 checks passed
@jnm2 jnm2 deleted the jnm2/tools2 branch September 9, 2025 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants