Loading App.config files of class library projects and using them when running tests in LocalTestRunner

[mcichonqa]

Hi, is it possible for the runner to run tests for the DLLs defined in TestPackage and use App.config from those DLLs?

I can't run my tests with specific configurations (App.config), i.e. coming from a specific class library project using the LocalTestRunner runner for executing.

I added vs-project-loader dll to the addins directory and it just works, because it downloads project settings like this:

var loader = node.ExtensionObject as IProjectLoader;
var condition= loader.CanLoadFrom("randompath\\ Akeneo.csproj");
IProject project = loader.LoadFrom("randompath\\ Akeneo.csproj");
TestPackage testPackage = project.GetTestPackage();

I'm not sure if testPackage contains information about a specific App.config from Akeneo.csproj?

For testing, I added a new App.config to my console application called NUnitTestRunner, in which I have:

<appSettings> <add key = "name" value = "ConsoleApp" /> </appSettings>

I also added App.Config to the Akeneo class library project:
<appSettings> <add key = "name" value = "ClassLibraryAkeneo" /> </appSettings>

When running tests in LocalTestRunner I do it like this:

LocalTestRunner runner = new LocalTestRunner(services, package);
runner.Load();
var result = runner.Run(_listener, filter);

While debugging in one of my tests, I check the configuration and expect a configuration from the Akeneo project.

var config = ConfigurationManager.AppSettings ["name"]. ToString ();

Config is "ConsoleApp" I don't know why.

Additionally, I tried to do it a different way. I used nunit-project-loader.dll and loaded a configuration file with the extension .nunit, but had the same result.

NUnit - 3.11.0, NUnit.Engine - 3.11.1
ConsoleApp(.NET Framework 4.6.1)
ClassLibrary(.NET Framework 4.6.1)

Can someone please help to find a solution?

[ChrisMaddock]

If I'm understanding correctly, I think what you're after should be default behaviour, but is currently affected by a bug. See #418 for some of the details - you're not using nunit proj files it seems, but I expect it's the same root cause.

Can you add some details of how your test package is structure, the code you're using to create it, and any other Package Settings you're using? Also, as a workaround, have you tried specifying the ConfigurationFile package setting, to force the engine to locate the correct app.config file?

[ChrisMaddock]

Hi @mcichonqa - I've moved your question over to this repo, as this is where the engine lives. Appreciate that's not clear from the repository names!

I've just realised we're talking about LocalTestRunner. This runs tests in the main app domain by definition, so would have the main app domains config, rather than any particular one for your test assembly. If you needed a separate config file for your test assemblies, you'd need to load them in separate app domains or processes, wouldn't you? Can you clarify why you need to use LocalTestRunner?

More generally, I'd also ask if there's a particular reason you're interacting with LocalTestRunner directly? The intended way to interface with the engine is through the nunit.engine.api assembly, and instead set appropriate package settings to have the engine run in the process/domain mode that you want to do. Some new docs have gone live recently which covers this a little: https://docs.nunit.org/articles/nunit-engine/Getting-Started.html

[mcichonqa]

Hi ChrisMaddock :)
Thanks for answer. Im executing my tests based on documentation(https://docs.nunit.org/articles/nunit-engine/Getting-Started.html) which you provided and all works fine(I can load my app configs from class library).

I used LocalTestRunner for debugging. In console application I have the code that runs the tests:

IProject project = projectLoader.LoadFrom(projectPath);
var package = project.GetTestPackage(_configName);
ITestEngine engine = TestEngineActivator.CreateInstance();
var runner = engine.Services.GetService<DefaultTestRunnerFactory>().MakeTestRunner(package);
runner.Run(new Listener(), filter);

The tests are invoked with the correct configuration, but... the console application and the class libraries with tests are referencing by a common class library project.
In the common class library project I have a static dictionary whose number of elements will change depending on how many tests are executed(each test adds something to the dictionary in SetUp phase). Using LocalTestRunner, I was able to get items from this dictionary in console application that runs the tests.

While executing tests with code:

IProject project = projectLoader.LoadFrom(projectPath);
var package = project.GetTestPackage(_configName);
ITestEngine engine = TestEngineActivator.CreateInstance();
var runner = engine.Services.GetService<DefaultTestRunnerFactory>().MakeTestRunner(package); 
runner.Run(new Listener(), filter); 

my static dictionary from the Common Class Library project is empty in console application that runs the tests.

Do you know how to solve this problem please?
Thanks

[ChrisMaddock]

Ahh, ok.

So, what you're relying on only works because you're running your tests in the same app domain as your test runner - so the tests and the test runner share the same static variables. By default, NUnit isolates your test into their own domains and/or processes, to make sure that no code in your test runner should ever affect the tests themselves - and your tests run in as close to a "clean" envrionment as possible.

What you're trying to do with a shared dictionary is going against that. If you really want to do this, you can add the PackageSetting DomainUsage with the value None. But I'd really recommend architecting the problem away instead - shared state between your test runner and tests is going against what the engine was built for!

Perhaps you can clarify exactly what you're using this dictionary for, and we might be able to make some suggestions? One option might be writing out whatever's in your dictionary via TestContext.Write(), and capturing the output with an Event Listener.

[mcichonqa]

These settings solved my problem.

TestPackage package = project.GetTestPackage(_configName);
package.AddSetting("DomainUsage", "None");
package.AddSetting("ProcessModel", "Single");

In general, the problem looks like this: before each test(in SetUp), I add to the static dictionary (test name, session), this enables logging of communication for each test, which is easier to find by the test ID, which is a session. It also allows me to extract a specific session from the static dictionary and use it to log in for Runner events.

You wrote about using ITestEventListener, so the session would have to be pulled from the console?
<output>session</output>

Currently in app console, Im using this:
string testCaseSessionId = SessionTestManager.GetTestSession(testCase.TestCaseName);<-- SessionTestManager is from Common

Thanks ;)

[mcichonqa]

@ChrisMaddock I would have two more questions, forgive me for the same thread.

1.What about .NET Core and Engine, it is possible to use engine (executed from .NET Framework console app) for .NET Core test projects? I had trouble loading .NET Core 2.1 projects with the ".csproj" extension. There were no files in this format.
2.How to load configuration in .NET Core? AppConfig is no longer used in .NET Core, but json files.

Thanks again for your help.

[ChrisMaddock]

Hi @mcichonqa - sorry it's taken me a while to get back to this one.

1.What about .NET Core and Engine, it is possible to use engine (executed from .NET Framework console app) for .NET Core test projects? I had trouble loading .NET Core 2.1 projects with the ".csproj" extension. There were no files in this format.

Not yet - this feature is tracked as #478

2.How to load configuration in .NET Core? AppConfig is no longer used in .NET Core, but json files.

This isn't yet implemented. If it was something you were interested in looking into further, we'd appreciate the contribution! 🙂

[mcichonqa]

Hi @ChrisMaddock ;)

My experience with configuration in .NET Core:
1.runsettings is good approach for configuration , but this also has limitations in .NET Core. If we want execute tests for several projects we can use only the same runsettings file which is global. This is not implemented microsoft/vstest#2418 , but open ;)
2.Deserialized JSON file as configuration in SetUpFixture.

If this task microsoft/vstest#2418 will be implemented, runsettings will be perfect ;)

[CharliePoole]

Also, to be clear, .runsettings is a feature of vstest and the VS IDE test window, not of nunit engine. Those runners have to translate any settings that come in from the file into settings that the engine understands.

[ChrisMaddock]

Charlie's right - runsetting's is a VSTest feature, not an NUnit one I'm afraid.

I'm not sure there's anything further to discuss here, so I'll close this issue now. Let us know if I've missed anything, and I can reopen. 🙂