Make Test conditions work in Visual Studio

[Wraith2]

When developing changes or investigating bugs in SqlClient it is important to be able to run the tests. This can be done from the commandline using complex syntax but most people prefer to be able to use the tests from inside visual studio. If try to run tests inside visual studio you will quickly discover that all tests run regardless of their conditions which means that even on a clean working master build you will wait a long time for test failures and have to sort through which failures are expected because they should not have been run.

A simple example is ConnectivityTest.ConnectionAliasTest. This is defined as:

        [PlatformSpecific(TestPlatforms.Windows)]
        [ConditionalFact(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsSQLAliasSetup))]
        public static void ConnectionAliasTest()

The test should not be executed unless it the IsSQLAliasSetup function returns true. I have not configured the alias, that function never returns true. The test is executed anyway.

I'd like to be able to run tests from inside visual studio and have any applied conditions correctly validated preventing tests that cannot succeed from being executed. Being able to evaluate whether the change i'm working on is breaking other parts of the library will enable much quicker testing allowing less reliance on the, no longer available to contributors, CI.

[ErikEJ]

@Wraith2 Have you tried this: xunit/xunit#2433 (comment)

[Wraith2]

I hadn't tried it. I have now. Makes no difference.

[MichelZ]

There's an arcade issue about this (the project which defines the ConditionalFact attribute)
dotnet/arcade#15207 (comment)

[Wraith2]

Is it an arcade issue somehow? If it cross-team co-op can still happen and get if fixed which is all I want.

[MichelZ]

I do believe it's an issue how the Attribute is coded. Here's another issue into that same direction: dotnet/arcade#15195 (So yes, IMHO it's an issue that can only be solved in the Arcade repo)

[David-Engel]

I never run all tests from the GUI because of this. Just targeted tests. Then I use the command line for broader validation. I find this annoying, too, as I'd rather just use the GUI. Would be nice if it got fixed.

[paulmedynski]

The way we specify, categorize, and filter tests is a big Quality of Life task on my plate. I will own this and drive discussions and changes to improve things going forward.

[akoeplinger]

I fixed the issue with ConditionalFacts not working in VS with dotnet/arcade#15667, once you upgrade to Microsoft.DotNet.XUnitExtensions version 10.0.0-beta.25175.1 or later it should start working.

The only other thing you need to do is add a .runsettings file that filters out the category=failing tests (which is what ActiveIssue and a couple other attributes use).