v3.3.0 (and v3.3.1) breaks test: Could not load file or assembly in method with AssemblyInitialize attribute

[Andreas-Dorfer]

I have the following test code that works with no issues up to v3.2.2, but fails with v3.3.0 and v3.3.1 (see MsTestRegression.zip for running example):

using FsCheck;

namespace MsTestRegression;

[TestClass]
public class TestClass
{
    [TestMethod]
    public void TestMethod1()
    {
        Prop.ForAll((IEnumerable<int> x) => true).QuickCheckThrowOnFailure();
    }

    [AssemblyInitialize]
    public static void TestInitialize(TestContext _)
    {
        Arb.Register<Arbitraries>();
    }
}

public class Arbitraries
{
    public static Arbitrary<IEnumerable<T>> EnumArb<T>() => new EnumerableArb<T>();
}

public sealed class EnumerableArb<T> : Arbitrary<IEnumerable<T>>
{
    public override Gen<IEnumerable<T>> Generator => Arb.Generate<List<T>>().Select(l => (IEnumerable<T>)l);
}

(I'm using FsCheck).

When I debug the test with v3.2.2, there are no exceptions. However, when I debug the test with v3.3.0 (or v3.3.1), there are two exceptions thrown.

First:

Second:

I think these exceptions cause the test to fail. It seems like something about loading assemblies in a method with the AssemblyInitialize attribute has changed with v3.3.0. The actual test method TestMethod1 then fails because Arb.Register<Arbitraries>() doesn't work as expected.

The test also fails with v3.3.0 when run from the CLI (but works with up to v3.2.2).

*) .net8
*) MSTest.TestFramework 3.3.1
*) Visual Studio 17.9.5

[MarcoRossignoli]

I'm getting different exception in VS and also in CLI(the same exception)

can you cleanup the solution removing all obj/bin folders?

Which version of VS are you using?

[Andreas-Dorfer]

I do get that exception too with v3.3.0. But that's a subsequent error, because the registration of the arbitrary in TestInitialize doesn't work. That's what the exception message is saying: It cannot find an arbitrary for IEnumerable<int>, although I'm registrating the arbitrary for IEnumerable<int> in TestInitialize. When you downgrade to v3.2.2, the arbitrary registration works again, and the test runs successfully.

To reproduce the other 2 exceptions, you have to enable all "Common Language Runtime Exceptions" and uncheck "Enable Just My Code".

[Andreas-Dorfer]

I don't know if it helps you with evaluating the issue, but I'm maintaining a project that integrates FsCheck with MsTest and I use MsTest + FsCheck in production. Upgrading MsTest to v3.3.0 breaks a lot of my production tests that depend on arbitrary registration (both locally and in the CI pipeline).

[Evangelink]

I will have a look next week but I recall having done some change to enable VSInstallationUtilities in different frameworks to add support for Assembly Resolution for netcore as requested by some company. It's possible this change has some side effect consequences.

[Evangelink]

I do confirm that with the repro and MSTest 3.2.2, the test pass while with 3.3.0 or 3.3.1 the test fails.

I have done some changes in local to try to revert #2110 and despite the few debugging exceptions all gone, I still end up with the same FsCheck exception: System.Exception: The type System.Collections.Generic.IEnumerable1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] is not handled automatically by FsCheck. Consider using another type or writing and registering a generator for it.`

With or without the changes, we are correctly able to call the content of the TestMethod1 so it would seem that the issue is on FsCheck. It's possible that some change in what is loaded causes FsCheck to fail but I don't know the logic they use.

I'd recommend creating a bug on their side so they can provide some more insights.

[osexpert]

I got the same\similar problem in 3.3.0-3.3.1.
It was a [TestMethod] with multiple [DataRow]. If I ran each [DataRow] individually, they worked. If I ran the [TestMethod], the first [DataRow] worked but the second one failed (weird COMException). So it's kind of a mystery error (race condition?).
I downgraded to 2.2.10.