Migration from specflow: Azure pipeline discovered test but incorrect number run

[cgiovenco]

Hello all, I recently migrated one of our specflow frameworks to reqnroll 2.2.1 using dotnet6. I got everything working as expected but when running the reqnroll framework via our existing azure pipeline I'm having issues and I don't know where to start troubleshooting this. When searching the web I mostly find issues related to not discovering test but I'm not having that problem. When executing everything is working as expected with building and executing but the issue is that the pipeline discovered number of test is the expected amount of test but the actual number of test being executed is about half of the discovered. For example, 525 test discovered and only 238 are now being executed. The pipeline was running all 525 test prior to converting to reqnroll so I'm unsure what would be causing the pipeline to be running less test than what was discovered if the only change was the framework migration. Any insight would be great.

[cgiovenco]

I reverted my code back to specflow and ran the same pipeline and all 525 test were picked up as expected. So still unsure what inside the rnr framework would be causing test from being discovered vs executed.

[melchiork]

And when do you run locally what are the results?
What .NET version is used in the pipeline?

[cgiovenco]

I don't understand what you mean by running locally. Locally via the tag I'm targeting in the pipeline visual studio test explorer identifies all 525 test. There's no .net version identified in the pipeline it's using the same yaml specflow/reqnroll. No change to the pipeline was made between just swapping from the framework upgraded from specflow to reqnroll.

[melchiork]

I assume the pipeline runs a dotnet test command, when you run the same command locally is the result identical to the pipeline?

[MacostaHitachi]

I had similar issue, and it was due to examples being translated into attributes, the rest of the tests are probably embedded within the 238 tests.
It also doesn’t work properly when Associating the test case.
You can switch to the old behavior (1 function per example)
By adding this to your reqnroll.json

"generator": {
      "allowRowTests": false
}

[cgiovenco]

Tried adding this but still had the same result via pipeline.

[tulavalle]

@cgiovenco

I'm having trouble running nunit via pipeline. I use Selenium Grid and we run the tests in an Azure pipeline in a Linux environment. It identifies all the tests, but when the first failure occurs, the execution is interrupted and the result is always success because it uses the number of tests that passed before the first failure as the total number of tests, always resulting in 100% success. I've already configured the pipeline task to continue even with an error, and I've included a .runtime file to continue if a test fails. In short, it works perfectly on Windows, but running the tests in nunit from a Linux machine seems to have some problem or require some additional configuration that I haven't identified yet.

For now, I've stopped using Nunit and returned to Xunit, which works both locally and via pipeline (Windows/Linux).

Has anyone else experienced this using NUnit in parallelism via Fixture or Children?

latest version of my csproj to test execution with nunit:

hooks.cs:

[tazambuja]

@cgiovenco , your tests run in nunit or xunit? If it is in nunit, could it not have some relation to the problem I reported? Does it use the same dependencies? In my case, it finds all the tests. The problem is that it stops the execution at the first failure and generates a result totaling 100% because it only considers the success scenarios before the first failure. I thought about the possibility of it being a bug in Reqnroll.NUnit...

[cgiovenco]

My test run in mstest