How to get IReqnrollOutputHelper in BeforeTestRun?

[Megadesty]

Related Documentation Page

automation/hooks.md

Type of the problem

Missing information

Problem Description

In #59 it says:

This means that some services that were available so far for these hooks are not available anymore (e.g. "IReqnrollOutputHelper"), let's see the impact of this.

I just migrated to Reqnroll (awesome work btw!) and I'm impacted by this. Before I used the SpecFlowOutputHelper in a BeforeTestRun hook to configure some global services which

• get registered with the objectContainer
• and print potential errors later while scenario execution

Is there any other way to get the output helper for global services?

[gasparnagy]

Not easy. I believe your solution was only working because you haven't used parallel execution (can you confirm?).

The challenge is that the test logging infrastructure provided by the test runners (mstest, nunit, xUnit) is bound to a test execution, so you cannot "remember" the output helper in a global service, because it would not know which test's logging infrastructure to be used. This is because if you run the tests parallel, there is no single "current" test, whose logging infrastructure could be used.

So basically what you did (because of the bug we fixed in #59) was picking the first execution worker thread and used the "current" test's logger within that worker thread.

If you would like to mimic the same behavior (that works only when you don't use parallel), you can do with a before scenario hook:

public static bool wasExecuted = false;
[BeforeScenario]
public static void ConfigureServices(ITestRunContext testRunContext, ITestThreadContext testThreadContext)
{
  if (wasExecuted) return;
  wasExecuted = true;

  var globalContainer = testRunContext.TestRunContainer; // this is the real global container used by all parallel threads
  var testThreadContainer = testThreadContext.TestThreadContainer; // this is what you have used in your hook
  //do service regs
}

For a real global service, all interfaces of your service should get the actual context and retrieve the output helper from it instead of storing it:

public class MyServiceClass {

  public void DoSomething(IScenarioContext context) {
    var outputHelper = context.ScenarioContainer.Resolve<IReqnrollOutputHelper>();
    //...
  }
}

[Megadesty]

Thank you very much for the detailed answer! You are right, the tests are executed sequentially. I guess it's the best to avoid logging in the way I did and maybe I can also activate parallel execution. Thanks again. I guess the docs doesn't need to be updated with this very specific topic so I close this issue.

[StefH]

@gasparnagy
When using this code, I get:
System.InvalidOperationException : No service for type 'Reqnroll.ITestThreadContext' has been registered.

[gasparnagy]

@StefH You mean the before scenario hook code? Can you try with TestThreadContext instead (so not the interface, but the concrete type).

[StefH]

Note that I'm using Reqnroll.Microsoft.Extensions.DependencyInjection and this is my code:

[Binding]
public class Startup
{
    private static IReqnrollOutputHelper? _outputHelper;

    protected Startup()
    {
    }

    [BeforeScenario]
    public static void Before(TestThreadContext testThreadContext)
    {
        if (_outputHelper != null)
        {
            return;
        }

        _outputHelper = testThreadContext.TestThreadContainer.Resolve<IReqnrollOutputHelper>();
    }

    [ScenarioDependencies]
    public static IServiceCollection CreateServices()
    {
        return new ServiceCollection()
            .AddLogging(builder => builder
                .AddProvider(new ReqnrollLoggerProvider(_outputHelper!)))
            .AddTransient<Calculator>();
    }
}

When using BeforeFeature or BeforeScenario, the CreateServices is called first, after that the Before method is called, so that does not work.

If I use BeforeTestRun, I get error:

System.Collections.Generic.KeyNotFoundException : The given key 'Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope' was not present in the dictionary.

The only code which does work is like:

[BeforeTestRun]
public static void BeforeTestRun(ITestRunnerManager testRunnerManager)
{
    if (_outputHelper != null)
    {
        return;
    }

    var runner = testRunnerManager.GetTestRunner();
    var context = runner.TestThreadContext;
    _outputHelper = context.TestThreadContainer.Resolve<IReqnrollOutputHelper>();
    testRunnerManager.ReleaseTestThreadContext(context);
}

Although I'm not sure this is the correct solution....

[gasparnagy]

@StefH which test execution fw (mstest, nunit, xunit) do you use?

[gasparnagy]

@StefH and do you (plan) to run your tests parallel?

[StefH]

• xunit
• sequential is ok for me now (I guess parallel will not work)

[gasparnagy]

@StefH So the problem is that with xUnit the IReqnrollOutputHelper works in a way that it forwards all log messages to the
xUnit ITestOutputHelper of the currently executing test. But in case of parallel execution the "currently executing" term does not makes sense (because there might be many of them). Therefore Reqnroll creates a separate IReqnrollOutputHelper instance for each separate test execution worker (the execution workers are represented by the ITestRunner interface), so in general you cannot have a "global" IReqnrollOutputHelper that you can store in a static fields or initialize a singleton logger for.

The proper solution would be (or should be) that the Reqnroll.Microsoft.Extensions.DependencyInjection plugin should call this [ScenarioDependencies] method (or a variation of it) for every single scenario where you are already in a scope of a scenario execution and therefore there is an IReqnrollOutputHelper already (it should pass this in as a param to the method). But as far as I see the plugin calls this method only ones, so the created ServiceCollection is a global singleton.

Of course, as you run your tests non-parallel, this thing could be patched somehow.

I think your workaround of calling testRunnerManager.GetTestRunner() will work in your case, because that will create a new test runner, but since you call the ReleaseTestThreadContext, it will give it back to the pool, so the scenario executions will receive the same runner and therefore the IReqnrollOutputHelper of that will always point to the output of the currently executed scenario.

You can make a quick test by saving the runner created by the [BeforeTestRun] hook to a static field and create a [BeforeScenario] hook with an ITestRunner parameter and compare that the two test runners are the same. If they are the same you are good to go.

Nevertheless, your case shows an interesting use-case that is not properly supported by the Reqnroll.Microsoft.Extensions.DependencyInjection plugin, so please create a separate issue (about how to access IReqnrollOutputHelper in the [ScenarioDependencies] method with this plugin) for this so that we can track and solve properly.

[StefH]

Thank you for this extended explanation.

The reason I ask this is because I'm creating a simple project which will implement a ILogger which uses this IReqnrollOutputHelper , so that I can forward the logging and see this in the test result.
(@gasparnagy : would this be something which can be included as a 3rd party nuget under the umbrella of Reqnroll?)

---

Nevertheless, your case shows an interesting use-case that is not properly supported by the Reqnroll.Microsoft.Extensions.DependencyInjection plugin, so please create a separate issue (about how to access IReqnrollOutputHelper in the [ScenarioDependencies] method with this plugin) for this so that we can track and solve properly.

I need to think on this how to correctly formulate this.

[gasparnagy]

I need to think on this how to correctly formulate this.

@StefH The reason statement you mentioned is just enough and please link your comment with the code example. It will mainly serve as a reminder for us that we don't forget, so no need to be super detailed.

[StefH]

@gasparnagy
OK, I did create a new issue. #317

Link to my project pull request is here:
StefH/Stef.Extensions.SpecFlow.Logging#1

(Maybe when this solution is accepted, I can move/donate my project into this repository?)

[gasparnagy]

@StefH That's cool! Are you using the ILogger only for the dependency injection or do you have other common use-cases as well when ILogger is needed/useful?

[StefH]

@StefH That's cool! Are you using the ILogger only for the dependency injection or do you have other common use-cases as well when ILogger is needed/useful?

The unit under tests uses a ILogger, so in that case the logging is forwarded to IReqnrollOutputHelper.

I'm intending to use the ILogger in my Reqnroll unit tests, in that case I'll just use IReqnrollOutputHelper to write some output.

I'll take a look if I can make a PR in this repository for this logger.

[StefH]

@gasparnagy
I created a initial PR for this new project, please take a look and tell me if the general direction is correct.
#321