How to make AfterScenario hook work properly

[tniemcewicz]

Hello all,
After our recent migration from specflow to reqnroll started having problems with my before/after hooks. We are using those hooks to initialize and close instance of our SeleniumExecutor which looks like:

    public static class SeleniumExecutor
    {
        [ThreadStatic]
        private static IWebDriver webDriver;
        (...)
        
        public static IWebDriver Initialize(BrowserType browser, string testName, DriverOptions options = null)
        {
            WaitTimeout = ConfigurationProvider.GetConfiguration().WebDriverSettings.WaitTimeout;
            webDriver = WebDriverFactory.InitializeDriver(browser, testName, options);

            return webDriver;
        }
        (...)
        
        public static void Close()
        {
            try
            {
                Driver().Quit();
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                throw;
            }
            finally
            {
                if (webDriver != null)
                {
                    Driver().Dispose();
                    webDriver = null;
                }
            }
        }
        (...)
   }

It looks like about 30% random tests are failing, during analysis we found that there are a lot of leftover instances of chrome.exe after each testrun, and failed scenarios were trying to execute their steps on browser instances left by previous scenarios.

our assemblyInfo is very simple

using NUnit.Framework;

[assembly: Parallelizable(scope: ParallelScope.Children)]
[assembly: LevelOfParallelism(2)]

same as reqnroll.json

{
  "nunitRetry": {
    "enabled": false,
    "tryCount": 1
  },
  "generator": {
    "addNonParallelizableMarkerForTags": [ "NonParallelizable" ]
  },
  "stepAssemblies": [ { "assembly": "Allure.ReqnrollPlugin" } ]
}

To confirm that it is a problem with After hook i added step calling SeleniumExecutor. Close at the end of each scenario and everything works perfectly fine, tests are green and there are no chrome.exe instances left.

How to properly set hooks to avoid such issue? I checked https://docs.reqnroll.net/latest/execution/parallel-execution.html and it looks like with such setup After/Before hooks should works as they were with specflow, but they are not.
thanks a lot in advance for any tip!

[clrudolphi]

Please share more information about how your SeleniumExecutor class is getting invoked. Are calls made to it from within Before/AfterTestRun hooks or other hooks?

I see from your posting that you appear to be using both a Retry plugin and Allure's plugin. Please rule them out as contributing here by running a quick test without them included.

[tniemcewicz]

I tried removing retry and allure from reqnroll.json leaving only

{
  "generator": {
    "addNonParallelizableMarkerForTags": [ "NonParallelizable" ]
  }
}

but tests are still failing and leaving chrome.exe instances.

Here are our hooks:

namespace UIAutomation.Tests.Hooks
{
    using System;
    using System.Threading.Tasks;
    using OpenQA.Selenium;
    using Selenium.Framework.Configuration;
    using Selenium.Framework.Utilities.Enums;
    using Selenium.Framework.Utilities.Helpers;
    using Reqnroll;
    using ConfigurationProvider = Configuration.ConfigurationProvider;

    [Binding]
    public class Hooks
    {
        private readonly ScenarioContext _scenarioContext;
        private readonly FeatureContext _featureContext;
        private readonly ITestRunner _testRunner;

        public Hooks(ScenarioContext scenarioContext, FeatureContext featureContext, ITestRunner testRunner)
        {
            _scenarioContext = scenarioContext;
            _featureContext = featureContext;
            _testRunner = testRunner;
        }

        [BeforeTestRun(Order = 0)]
        public static void BeforeTestRun()
        {
            SeleniumReporter.ClearScreenshotsDirectory();
        }

        [BeforeScenario(Order = 0)]
        public void BeforeScenario()
        {
            SeleniumExecutor.Initialize(ConfigurationProvider.GetConfiguration().WebDriverSettings.BrowserType,
                _testRunner.ScenarioContext.ScenarioInfo.Title);
        }

        [BeforeScenario(Order = 100)]
        [Scope(Tag = "Sequential")]
        public async Task BeforeScenarioSequential()
        {
            if (EnvironmentVariableProvider.GetEnvironment == TestEnvironment.Pipeline.ToString())
            {
                await WiremockHelper.ResetWiremockJournal();
            }
        }

        [AfterStep]
        public void AfterStep()
        {
            if (_scenarioContext.TestError != null)
            {
                try
                {
                    // SeleniumReporter.TakeScreenshot(CreateScreenshotFileName(_scenarioContext, _featureContext));
                    SeleniumReporter.TakeFullPageScreenshot(CreateScreenshotFileName(_scenarioContext, _featureContext));
                }
                catch (WebDriverException e)
                {
                    Console.WriteLine(e);
                }
            }
        }

        [AfterScenario()]
        public async Task AfterScenario()
        {
            if (ConfigurationProvider.IsRemoteDriverInUse())
            {
                if (_scenarioContext.TestError == null)
                {
                    BrowserStackReporter.UpdateTestStatus(TestStatus.Passed);
                }
                else
                {
                    BrowserStackReporter.UpdateTestStatus(TestStatus.Failed, _scenarioContext.TestError.Message);
                }

                await BrowserStackReporter.PrintReportUrl();
            }

            SeleniumExecutor.Close();
        }

        private static string CreateScreenshotFileName(ScenarioContext scenarioContext, FeatureContext featureContext)
        {
            var featureTitle = featureContext.FeatureInfo.Title;
            var scenarioTitle = scenarioContext.ScenarioInfo.Title;
            return $@"{featureTitle}-{scenarioTitle}";
        }
    }
}

and SeleniumExecutor class

namespace Selenium.Framework.Configuration
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using OpenQA.Selenium;
    using OpenQA.Selenium.Interactions;
    using OpenQA.Selenium.Remote;
    using OpenQA.Selenium.Support.UI;
    using Utilities.Enums;

    public static class SeleniumExecutor
    {
        [ThreadStatic]
        private static IWebDriver webDriver;

        private static IDictionary<string, Uri> pages;

        public static ISearchContext SearchContext => Driver();

        public static IJavaScriptExecutor JavaScriptExecutor => (Driver() as IJavaScriptExecutor)!;

        public static Actions Actions => new(Driver());

        public static IDictionary<string, Uri> Pages => pages ??= GetPagesFromConfig();

        private static int WaitTimeout { get; set; }

        public static IWebDriver Driver(DriverOptions options = null)
        {
            return webDriver ?? Initialize(ConfigurationProvider.GetConfiguration().WebDriverSettings.BrowserType, null, options);
        }

        public static IWebDriver Initialize(BrowserType browser, string testName, DriverOptions options = null)
        {
            WaitTimeout = ConfigurationProvider.GetConfiguration().WebDriverSettings.WaitTimeout;
            webDriver = WebDriverFactory.InitializeDriver(browser, testName, options);

            return webDriver;
        }

        public static string GetRemoteSessionId()
        {
            var remoteDriver = (RemoteWebDriver)Driver();
            return remoteDriver.SessionId.ToString();
        }

        public static bool IsDriverInitialized() => webDriver != null;

        public static void Close()
        {
            if (webDriver == null)
            {
                return;
            }

            try
            {
                // Driver().Close();
                Driver().Quit();
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                throw;
            }
            finally
            {
                if (webDriver != null)
                {
                    Driver().Dispose();
                    webDriver = null;
                }
            }
        }

        public static void PrintLogs()
        {
            var logs = Driver().Manage().Logs.GetLog(LogType.Performance);
            foreach (var log in logs)
            {
                Console.WriteLine(log.Message);
            }
        }

        public static WebDriverWait WaitDriver(int customTimeout = 0, Type[] exceptionsToIgnore = null)
        {
            var waitTimeout = customTimeout == 0 ? ConfigurationProvider.GetConfiguration().WebDriverSettings.WaitTimeout : customTimeout;
            var waitDriver = new WebDriverWait(Driver(), TimeSpan.FromSeconds(waitTimeout));
            if (exceptionsToIgnore != null)
            {
                foreach (var type in exceptionsToIgnore)
                {
                    waitDriver.IgnoreExceptionTypes(type);
                }
            }

            return waitDriver;
        }

        public static WebDriverWait WaitDriver(TimeSpan? customTimeout = null, Type[] exceptionsToIgnore = null)
        {
            var waitTimeout = customTimeout ?? TimeSpan.FromSeconds(ConfigurationProvider.GetConfiguration().WebDriverSettings.WaitTimeout);
            var waitDriver = new WebDriverWait(Driver(), waitTimeout);
            if (exceptionsToIgnore != null)
            {
                foreach (var type in exceptionsToIgnore)
                {
                    waitDriver.IgnoreExceptionTypes(type);
                }
            }

            return waitDriver;
        }

        private static Dictionary<string, Uri> GetPagesFromConfig()
        {
            return ConfigurationProvider.GetConfiguration().Pages.ToDictionary(page => page.Name, page => new Uri(page.Path));
        }
    }
}

[ajeckmans]

I don't think there is an issue in your code. As I remember Nunit creates a couple of test threads and a test is guaranteed to run on the same thread. When waiting (with async code) no other test is scheduled on that thread, so you should be good with the static usage. Seems like the afterscenario is not always called. Can you double check by writing to a log somewhere that for each beforescenario you're actually also hitting an afterscenario in that same thread?

I do want to note that the statics will prevent you from ever switching to a different test framework such as xunit, mstest, tunit or our own when it inevitably is written. Most test frameworks will schedule tasks on whatever test thread happens to be free, simply because they do use the default thread pool unlike Nunit. You're better off relying on the DI container in that case to dispose of your objects.