I feared this curve ball would show up in the issue list some day.

Under the covers, the extension runs:

• Coverlet for .net core projects
  https://github.com/OpenCover/opencover/wiki/Usage
• OpenCover for .net framework projects
  https://github.com/opencover/opencover/blob/master/main/OpenCover.Documentation/Usage.pdf

Unfortunately, neither allows for fine-grained single (or a few) test execution/coverage scenarios.
We could ask the developers of those projects to add that feature but I'm not optimistic we would get that any time soon if at all.

I sincerely apologize for this. I don't think i can deliver this feature.
Still, please post the issues there; link back to this one and maybe we'll get a positive response (they are aware of this project; they may just throw us a bone)

@FortuneN When running vstest.console via opencover you can pass /TestName:TestMethod1 to vstests, via opencover -targetargs switch

Thanks @alex-jitbit

I figure i can also use module and class filters (which would be compatible for both openCover and coverlet down to single class level; i'm hoping to always be able to support the same features for both)

The missing part of this puzzle is "how do i know which tests were run".
My visual studio extension development knowledge is still low and my preliminary research has been unfruitful.
Can you help?

FYI: I'm reopening the issue with this renewed interest.

@Pompeoar You could just disable the addin until you need to get coverage.

@tonyhallett That's pretty much how I've worked around it.

@tonyhallett Maybe we could provide a simple Enable/Disable toggle on the Fine Code Coverage display window. This is far from the required solution but it could be a convince to be able to easily switch.

@FortuneN I was going to suggest that. As I said to you I have an idea. For coverlet it would mean lo longer running coverage after tests have run. The collector driver facilitates this. Fine Code Coverage would need to just watch the TestResults folder for the emitted cobertura file. The difficulty is knowing this path as it is specified in .runsettings and is not available by reflection from test container. There is an exported internal class that provides this information for each test container.
The run settings are available to a test adapter implementing ITestDiscoverer and retrieved in discovery. Through refection the setter could be used to change these settings to our advantage.

Getting these run settings is necessary to replicate Vs regardless of the collector. This is not currently implemented.

An alternative is just a button and textbox for runsettings path and then the TestResults path and the Data collector can easily be added by Fine Code Coverage.

After Christmas I can provide further details

In addition I think that when tests have finished it is possible to get stats on tests passed failed skipped through reflection.

If not using the data collector and having to re-run tests these values could be used to determine if coverage should be run.

get stats on tests passed failed skipped through reflection

It should be possible in one form or another.
If you could provide me with an article, sample code, ... then I can start crafting the fix for this issue

The collector driver facilitates this

If i'm not mistaken, i explored the data collector approach when i started.
What i did not like is the need to add a package reference to get that to work.

 <PackageReference Include="coverlet.collector" Version="x.x.x">

Don't get me wrong, I'm currently using that to generate and publish coverage reports for my Azure DevOps builds.
The choice I made is to avoid the requirement that a user must first adjust their project for FCC to work.
The way it is now. You install the extension, run tests and it just works (no muss, no fuss).

The package reference provides the collector dll and an msbuild target that makes the path to the collector available to the test framework so you only have to pass the friendly name.
There is something for deterministic builds as well.

Aside from deterministic builds there is no need for the package reference or msbuild changes. As long as we provide the DLL and can provide the path. Path can be a command line argument or in the runsettings. The latter is preferable.

The advantages of the collector is that there is no need to repeat the tests and as is mentioned on the coverlet site only the data collector is free from known issues that users of Fine Code Coverage could encounter. So no muss no fuss could have fuss.

In fact coverage is an opt in feature and perhaps this is how Fine Code Coverage should work. Just add a button as per https://docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested?view=vs-2019#analyze-code-coverage.

Discuss further in a few days.

Happy Christmas

I am are of that. That's the case with:

• Built in coverage (cited above)
• DotCover
• AxoCover (this one goes further and defines it's own test explorer)
• e.t.c

FCC goes in a different direction by choice.

I started FCC after seeing the magic of Live Unit Testing and the ultimate dream for FCC is a user experience close to that (for code coverage).
Note also that Code Lens shows in the source file how many tests are passing/failing per method and it does this in the background.

The user does not have to wait for the coverage process to end unless they actually want to see the results.
This is why FCC is a none-blocking background thread that gets stopped by a subsequent build.
I believe this perception can be addressed by better UI (in the short term, i could print out that suggestion).
The idea is, you should just run your tests as you normally do and check coverage only when you are interested.
Also, this is why i do not auto-switch to the 'Fine Code Coverage' tab/window after coverage completes (because i do not assume you want to see it every time)
I believe logging to the Build output is what breaks this perception and makes users believe they must wait for the coverage to complete every time.

Anyone who has ui tests will never want the full suite run automatically each time a single test is run and will have to switch on which is not much different to a button.

Running a background thread for a large test suite would have a impact on Vs.

With regards to runsettings. Anyone who uses runsettings data in their tests will currently not be able to use the addin at all.

Anyone who has ui tests will never want the full suite run automatically each time a single test is run and will have to switch on which is not much different to a button

We will need to get to point where we have high efficiency (i.e. run/re-run only what needs to be re-run).
That starts with solving this issue.

Also, we can (in the future) include the ability to limit what gets run automatically.
Perhaps have profiles with different run configurations; the default being the automatic one.
The other profiles could be run by 'a button'. (I'm not opposed the idea)

Running a background thread for a large test suite would have a impact on Vs.

Currently, that's handled by

• running a separate thread to avoid blocking VS
• running CLIs instead of in-process libraries so that processing happens in external processes outside of VS

(I know coverlet and report-generator are runnable in-process as libraries; i never checked OpenCover coz by the time i integrated it, i had already made the choice to run CLIs)

I intend to move the code under the 'Core' folder into an external micro-web-server process.
That folder contains all the generic none-VS stuff (by design)
This would put us in a position where we can have

• Fine Code Coverage as standalone CLI
• Fine Code Coverage in other IDEs that we can write extensions for e.g. vs code.

I got the idea from Omnisharp.
FCC is going to be cross-platform, cross-ide and even ide-less

With regards to runsettings. Anyone who uses runsettings data in their tests will currently not be able to use the addin at all.

I'm currently using runsettings in my work projects; I don't have a problem.
Maybe i misunderstand what you mean (maybe attach a repro).

Speaking of dreams, one of mine is to get listed on dotnet-foundation list
https://dotnetfoundation.org/projects

The coverlet targetargs does not have run settings. Simple as. No repro required.

Anyone who uses runsettings data in their tests will currently not be able to use the addin at all.

I agree.
FCC will still run but ignore your runsettings entirely which is catastrophic because your tests need the runsettings.
(The way i use runsettings seems to make me immune to this but that's besides the point. You are right)

.NET CORE

The coverlet targetargs does not have run settings.

It does.
Consider the following CLI command

coverlet /path/to/test-assembly.dll --target "dotnet" --targetargs "test /path/to/test-project --no-build"

• --target "dotnet"      : refers to the dotnet CLI https://docs.microsoft.com/en-us/dotnet/core/tools
• --targetargs "test ..." : refers to the dotnet test command https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test

Thus you can apply all the args that are possible for dotnet test and they include '--settings'

-s|--settings <SETTINGS_FILE>

NET FRAMEWORK

For .net framework we can have runsettings support through the vs test console (MsTestPlatform in FCC)
https://docs.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019

/Settings:[file name]

SOOOOOOO

We can do the runsettings fix for both .net core (coverlet) and .net framework (opencover).
The only missing piece is detecting which runsettings file was used (I use multiple run settings per solution so we can't just scan the file system). I suspect this is somewhere in the "OperationState_StateChanged" method parameters (sender, e).

We should span a new issue for runsettings support.
Do you want work on that or should i?

It does

Your args do not. I am fully aware that they can !

As I have already said this information does not appear to be available with reflection in Operation__StateChanged but is available if you create your own test adapter. The question is - if you create a fake adapter just to catch the run settings in discovery is that sufficient to be always notified of the current settings.

In a couple of days when I return to work I can provide what I have found so far in terms of documentation and internal classes. We can then discuss approaches.

We could just add to Vs options additional testargs.

I still think that it should be possible to use the collector.

As for the other points that you raised e.g only running tests for changed code we can discuss that too. For that you would need to use Roslyn.

I will add an issue tomorrow morning on runsettings and add the information that I found. If you wish to tackle it then proceed as I will not be able to look at it for a couple of days. If you can wait those days I can spare some hours to fix it assuming the information is available.

I found this

e.Operation.Configuration.UserRunSettings.LastRunSettingsFilePath

Yes but that's no good. There is a solution method there as well but looking at the source I could not see how it could be a runsettings that is from the user choice. If it was possible then you could read the project file to see if it specified runsettings.

See https://github.com/microsoft/vstest-docs/blob/master/RFCs/0004-Adapter-Extensibility.md for Idiscoverycontext

https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.ObjectModel/Adapter/Interfaces/IDiscoveryContext.cs

During my research when i started the project, i found this article :
https://devblogs.microsoft.com/devops/writing-a-visual-studio-2012-unit-test-adapter
I abandoned this approach when i realized i didn't need it at the time. Perhaps it's time to revisit.

Yes but that's no good

Why?

Well I think it will be no good. Because of Last. What do you get first ? I have read the source with Reflector and ruled it out. Do you have Reflector or DotPeek ? In the code add a line to get the assembly location and then decompile in one of them.

I abandoned this approach when i realized i didn't need it at the time. Perhaps it's time to revisit.

Yes that is the test adapter that I am referring to.

Another spanner to throw in the works. Is the reflection code safe as is ? Is it possible for custom test adapter to discover containers that are not the type you are expecting ?

If you can find the solution runsettings method - sorry don't have the details as not at my laptop you could test if the path is correct when changing run settings in vs.

As I said before I have found an internal class that seems to have the logic that is required for runsettings on a project basis. I can send that tommorow.

I did try the custom adapter approach - as an asset in the vsix. The details are in the link above I think. Unfortunately could not get it to be loaded in the short time that I tried.

See if you can get it to work. You can check Nunit if you want to compare your setup.

I wasn't aware of project level run settings; All of mine are solution level.

A quick google search turned up the following link.

https://developercommunity.visualstudio.com/content/problem/151608/ability-to-set-runsettings-in-project-file.html

... in addition to specifying a runsettings file as in previous versions, there are now two more ways to do so.

1. Add a runsettings file named ".runsettings" at the root of the solution. 
   Visual Studio will then automatically pick this file up and use it in all test runs. 
   This applies to all tests run.
   [This feature is currently disabled by default. To enable it, you can turn it on by going to Test > Configure Run Settings > Auto Detect runsettings Files]
2. Specify a project level runsettings file through the build property `RunSettingsFilePath`.
   Visual Studio will always use the file in test runs for the specified project(s).
   This overrides any other runsettings file selection.

The first is what i've been using.
I'll check the second one. Seems that it would be easy to read that from the file.

Another spanner to throw in the works. Is the reflection code safe as is ? Is it possible for custom test adapter to discover containers that are not the type you are expecting ?

I don't know

This ms docs article confirms the 3 approaches
https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019

There is a Microsoft page dedicated to run settings. Auto detect solution, choose run settings for solution or project level in project file.

As for reading from the project file. Easy to read if the user follows the docs but they could use any msbuild property they like as part of the path and then no.

https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019
?

Yes that is it.

If you don't want to use Reflector or DotPeek the source is available on GitHub. For instance the implementation of IRunSettings https://github.com/microsoft/vstest/blob/603d9bc4bbd10bdffedc1ce0e15f978dc157c32f/src/Microsoft.TestPlatform.Common/RunSettings.cs

I have dotPeek (stand-alone); I'm not sure how to use it the way you've described.
Some instructions maybe

(I haven't used Reflector since it stopped being free)

Not at laptop so cannot help much.

Get the type of UserRunSettings e.Operation.Configuration.UserRunSettings.LastRunSettingsFilePath
and the assembly location. DotPeek should allow you to open that assembly by path to view source

Or search the GitHub repo once you have the type name.
That's it from me today. Let me know where you get to.

public string LastRunSettingsFilePath => this.hostState.Value.IsOpened ? this.runSettings.Data.LastRunSettingsFilePath : (string) null;

Don't answer now! lol

There is a method that should provide the solution settings. Cannot remember the name though. Have you checked the methods ? I have the details on my laptop if you can wait until tomorrow

I think I have written the necessary code. Testing it now, so hold back on doing any further work

Ok it works and it is with reflection so you are right that it was achievable although we may not have found the necessary code if I had not used Reflector on C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Microsoft.VisualStudio.TestWindow.Core.dll.

As I said before there was a class that provided the logic that we required - ContainerRunSettingsProvider.

Here is the reflection version of that code.

internal class RunSettingsRetriever
    {
		private object userSettings;
		public async Task<string> GetRunSettingsFileAsync(object userSettings,object testContainer)
        {
			this.userSettings = userSettings;
			var runSettingsFile = GetDefaultRunSettingsFilePath();
			var projectRunSettingsFile = await GetProjectRunSettingFileAsync(testContainer);
            if (!string.IsNullOrEmpty(projectRunSettingsFile))
            {
				return projectRunSettingsFile;
            }
			return runSettingsFile;
		}
		private string GetAndUpdateSolutionRunSettingsFilePath()
        {
			return userSettings.GetType().GetMethod("GetAndUpdateSolutionRunSettingsFilePath", BindingFlags.Public | BindingFlags.Instance).Invoke(userSettings, new object[] { }) as string;

		}
		private string LastRunSettingsFilePath()
        {
			return userSettings.GetType().GetProperty("LastRunSettingsFilePath", BindingFlags.Public | BindingFlags.Instance).GetValue(userSettings) as string;
		}
		private bool AutomaticallyDetectRunSettings()
        {
			return (bool)userSettings.GetType().GetProperty("AutomaticallyDetectRunSettings", BindingFlags.Public | BindingFlags.Instance).GetValue(userSettings);
		}
		private string GetDefaultRunSettingsFilePath()
		{
			string settingsFilePath = GetAndUpdateSolutionRunSettingsFilePath();
			var lastRunSettingsFilePath = LastRunSettingsFilePath();
			if (!string.IsNullOrEmpty(lastRunSettingsFilePath))
				return lastRunSettingsFilePath;
			if (!AutomaticallyDetectRunSettings() || string.IsNullOrEmpty(settingsFilePath))
				return (string)null;
			return settingsFilePath;
		}
		private static async Task<string> GetProjectRunSettingFileAsync(object container)
		{
			var projectDataProperty = container.GetType().GetProperty("ProjectData");
			if (projectDataProperty != null)
			{
				var projectData = projectDataProperty.GetValue(container);
				var projectRunSettingsFile = await (projectData.GetType().GetMethod("GetBuildPropertyAsync", BindingFlags.Instance | BindingFlags.NonPublic).Invoke(projectData, new object[] { "RunSettingsFilePath", (string)null}) as Task<string>);
				return projectRunSettingsFile;
            }
			return null;
		}

	}

So usage

if (e.State == TestOperationStates.TestExecutionFinished)
				{
					//.......
					var testConfiguration = (operationType.GetProperty("Configuration") ?? operationType.GetProperty("Configuration", BindingFlags.Instance | BindingFlags.NonPublic)).GetValue(e.Operation);
					
					var userRunSettings = testConfiguration.GetType().GetProperty("UserRunSettings", BindingFlags.Instance | BindingFlags.Public).GetValue(testConfiguration) as dynamic;
					//...

					var runSettingsRetriever = new RunSettingsRetriever();
					
					foreach (var container in testContainers)
					{
						
var project = new CoverageProject();
// You can decide how to deal with the async Task.
project.RunSettingsFile = runSettingsRetriever.GetRunSettingsFileAsync(userRunSettings, container).GetAwaiter().GetResult();


						

I will leave the full implementation of the code to you ! - i.e how to deal with the async file retrieval and the passing of the run settings file to Coverlet / OpenCover.

It has been tested with the auto detect run settings, a specific solution run settings and also with run settings specified in the project file !

Of course all of this reflection is liable to break. microsoft/vstest#1830

Of course all of this reflection is liable to break. microsoft/vstest#1830

I'm sure it will at some point during a VS version increase.
"The answer is don't think about it" ~ Rick
https://getyarn.io/yarn-clip/9657326a-aa9e-4e88-adaf-17ca50833e22
In the mean time we have happy users (albeit blissfully unaware of the that nightmares you and i are having behind the scenes)

... Ok it works ...
... it was achievable ...
... said before there was a class that provided the logic that we required - ContainerRunSettingsProvider ...
... Here is the reflection version of that code ...
... It has been tested with the auto detect run settings, a specific solution run settings and also with run settings specified in the project file !

!!!!!!!!!!!!!!!!!!! YOU ARE THE MAN !!!!!!!!!!!!!!!!!!!
👍 💯 🥇

I will leave the full implementation of the code to you ! - i.e how to deal with the async file retrieval and the passing of the run settings file to Coverlet / OpenCover

I'll start crafting.
...
DONE AND RELEASED!

@FortuneN you may be interested in this as a method of not running coverage combined with user options.

class TestRunResponse
    {
		public long FailedTests { get; private set; }
		public long PassedTests { get; private set; }
		public long SkippedTests { get; private set; }
		public long TotalTests { get; private set; }
		public bool IsAborted { get; private set; }
		public TestRunResponse(object response)
        {
			var responseType = response.GetType();
			FailedTests = (long)responseType.GetProperty(nameof(FailedTests), BindingFlags.Instance | BindingFlags.NonPublic).GetValue(response);
			PassedTests = (long)responseType.GetProperty(nameof(PassedTests), BindingFlags.Instance | BindingFlags.NonPublic).GetValue(response);
			SkippedTests = (long)responseType.GetProperty(nameof(SkippedTests), BindingFlags.Instance | BindingFlags.NonPublic).GetValue(response);
			TotalTests = (long)responseType.GetProperty(nameof(TotalTests), BindingFlags.Instance | BindingFlags.NonPublic).GetValue(response);
			IsAborted = (bool)responseType.GetProperty(nameof(IsAborted), BindingFlags.Instance | BindingFlags.NonPublic).GetValue(response);
		}
    }

if (e.State == TestOperationStates.TestExecutionFinished){
  var operationType = e.Operation.GetType();
  var testRunResponse = new TestRunResponse(operationType.GetProperty("Response",BindingFlags.Instance|BindingFlags.Public).GetValue(e.Operation));

// user could state don't run coverage if any failed or if number of tests > ....?					

you may be interested in this as a method of not running coverage combined with user options.
user could state don't run coverage if any failed or if number of tests > ....?

That would a nice feature to have.
I've created new issue for that

@Pompeoar
The next release has a visual studio option RunWhenTestsExceed. If you set this to 1 then running a single test will not run coverage.

Soon there will be a way to turn off auto coverage from the tool window.