MSTest v3 Breaking Changes

[Evangelink]

Description

The goal of this ticket is to announce the upcoming breaking changes in MSTest v3.

Implemented

• Follow Microsoft .NET lifecycle:
  • Dropped support for .NET Framework 4.5 (moved to 4.6.2)
  • Dropped support for .NET Standard 1.0 (moved to 2.0)
  • Dropped support for UWP before 16299
  • Dropped support for WinUI before 18362
  • Dropped support for .NET 5 (replaced with netcoreapp3.1 and .NET 6)
• Merge timeout behaviours for .NET Core and .NET Framework Merge timeout behaviors for .NET Core and .NET Framework #1296
• Refactor available ctors for DataRowAttribute Refactor available ctors for DataRowAttribute #1332
• Introduce strategies for test ID generation Introduce strategies for test ID generation #1286
• Remove Assert.AreEqual/AreNotEqual overloads with object object Remove Assert.AreEqual/AreNotEqual overloads with object object #1429
• AreSame/AreNotSame use generic instead of object Assert.AreSame/AreNotSame use generic instead of object #1430
• Unify DeploymentDirectory location across target frameworks Unify DeploymentDirectory location across target frameworks #1414
• Add class/assembly cleanup/init messages to first/last test Add class/assembly cleanup/init messages to first/last test #1390
• Unwrap real exception from TargetInvocationException Unwrap real exception from TargetInvocationException #1254

[adcloeshelle]

@Evangelink - Do you have a rough timeline on when these changes will be released? My team is particularly excited about #1286 so we can upgrade to the latest versions while keeping the old Test ID behavior. Thanks for any information you can provide!

[Evangelink]

Hi @adcloeshelle, sadly we don't have any estimate at the moment. The ticket you are mentioning will fix a lot of issues in the backlog so we are actually reducing the scope of v3 so we can ship it as fast as possible. When I manage to get some release date I will ping here and set some date in the milestone too.

[Evangelink]

@adcloeshelle We have published some pre-release version if you want to test it out and confirm this is fixing issues for you.

[adcloeshelle]

Thanks! I work with @Cjewett and we will test this out.

[jimmylewis]

Probably worth calling out explicitly: #1332 mentioned above breaks existing tests that have more than 16 arguments passed via DataRow. (Yes, this was a real issue while upgrading 😭).

[Evangelink]

Hey @jimmylewis, sorry, I will update the release note to make it more explicit. Out of curiosity how many parameters are you passing? You can "fix" this change by wrapping your parameters in some array.

To give you a bit of context on this change. The older MSTest were using object, params object[] signature which was causing issue when trying to pass 2 arrays. Looking at xUnit and NUnit it seems they are using params object[] signature but doing so it's becoming impossible to pass string array as argument (compiler error). After some investigation, we decided to follow the pattern used by runtime for Action (with 16 overloads). Looking up online on GitHub we didn't see many repositories using this many parameters and thought it was "relatively" safe.

[jimmylewis]

We had about a half-dozen cases of 17 and 18 parameters. I was able to refactor them down to 16, but it wasn't clear at first why they'd suddenly stopped building other than the error that DataRow didn't have a constructor taking 18 arguments.

Thanks for making the note clearer, hopefully my team is an outlier and few others encounter this. 😁

[cremor]

@Evangelink FYI, the changelog contains a few breaking changes that are not mentioned here:

[breaking change] Unify DeploymentDirectory location across target frameworks #1414
[breaking change] Add class/assembly cleanup/init messages to first/last test #1390
[breaking change] Unwrap real exception from TargetInvocationException #1254
[breaking change] Remove Assert.AreEqual/AreNotEqual overloads with object object #1429

The last one is especially important because of #1550

There is also one the other way around. This issue mentions #1431, but that is not marked as a breaking change in the changelog (where it is linked as #1430).