exception occurred while invoking executor Unexpected token '(' at position 100 in selection expression.

[livebe01]

I have the following test

[Test]
[TestCase(ThisEnum.Option1, "ABC")]
[TestCase(ThisEnum.Option2, "DEF")]
[TestCase(ThisEnum.Option3, "GHI")]
public void TestSomething(ThisEnum thisEnum, string expected)
{
// do something
}

I can run the tests individually, and as a group in Rider w/o issue. But, for some use cases, it'd be really useful to be able to run parameterized tests individually, from the CLI.

I'm developing on a Mac using Rider. My project is using "NUnit" Version="3.13.3" and "NUnit3TestAdapter" Version="4.3.1".

I run the following from the Mac terminal:
dotnet test -- NUnit.Where="test==Some.Name.Space.SomeClass.TestSomething(EnumValue,\"ABC\")"

and get this error:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Unexpected token '(' at position 100 in selection expression.
Stack trace:
   at NUnit.Engine.TestSelectionParser.Expect(Token[] valid)
   at NUnit.Engine.TestSelectionParser.Parse(String input)
   at NUnit.Engine.TestFilterBuilder.GetFilter()
   at NUnit.VisualStudio.TestAdapter.NUnitTestFilterBuilder.FilterByWhere(String where) in C:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestFilterBuilder.cs:line 93
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in C:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 129
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionDecorators.SerialTestRunDecorator.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

When I run dotnet test -- NUnit.Where="test==Some.Name.Space.SomeClass.TestSomething", all three tests run w/o error. I wonder if you can point me at what I might be doing wrong or what I might try to get passed this.

Thanks!

[stevenaw]

Hi @livebe01
I don't have a Mac myself to test this, but can you try escaping the ( and ) by putting a back slash in front? For example:

dotnet test -- NUnit.Where="test==Some.Name.Space.SomeClass.TestSomething\(EnumValue,\"ABC\"\)"

[livebe01]

Thanks @stevenaw, I tried your suggestion, but it results in the same error/behavior.

It's strange to see C:\repos\nunit\nunit3-vs-adapter\src referenced in the trace. That's a Windows path.

[OsirisTerje]

The same error happens on Windows. It should work according to the docs https://docs.nunit.org/articles/nunit/running-tests/Test-Selection-Language.html#simple-expressions

The error happens in the NUnit.Engine, so I'll transfer the issue there.
Repro code here: https://github.com/nunit/nunit.issues/tree/main/Issue4880

[mikkelbu]

I took a quick look at this when it was reported and as far as I quickly could determine then

nunit-console/src/NUnitEngine/nunit.engine/Services/TestSelectionParser.cs

Line 16 in d238d33

private static readonly Token LPAREN = new Token(TokenKind.Symbol, "(");

is only used for expressions inside parenthesis and not for arguments (and this is somewhat confirmed by looking at the tokenizer - https://github.com/nunit/nunit-console/blob/d238d33023c1ffe96b2c6bf44f2077569c17e763/src/NUnitEngine/nunit.engine/Services/Tokenizer.cs#L86C30-L86C47 ).

I'm actually unsure if this have ever worked (perhaps the documentation and code is out of sync?).

Ps. NUnitLite seems to have the same problem

[CharliePoole]

If this is an error in the engine, we should be able to repro it with a failing unit test for the tokenizer and/or the parser. I'll try to do that when I get back to work in a few weeks but if somebody wants to start on it now, please go ahead. We haven't often checked in failing tests deliberately, but that's actually a good way to prove a bug and does no harm since we won't merge a failing branch.

It's harder for us if the failure only occurs on a Mac, but @livebe01 could give it a try as well. We run into this problem (lack of devs with Macs on the team) periodically.

[livebe01]

I took a quick look at this when it was reported and as far as I quickly could determine then

nunit-console/src/NUnitEngine/nunit.engine/Services/TestSelectionParser.cs

Line 16 in d238d33

private static readonly Token LPAREN = new Token(TokenKind.Symbol, "(");

is only used for expressions inside parenthesis and not for arguments (and this is somewhat confirmed by looking at the tokenizer - https://github.com/nunit/nunit-console/blob/d238d33023c1ffe96b2c6bf44f2077569c17e763/src/NUnitEngine/nunit.engine/Services/Tokenizer.cs#L86C30-L86C47 ).
I'm actually unsure if this have ever worked (perhaps the documentation and code is out of sync?).

Ps. NUnitLite seems to have the same problem

Per nunit/docs#316, it looks like the feature may have worked at some earlier point.

[livebe01]

It's harder for us if the failure only occurs on a Mac, but @livebe01 could give it a try as well. We run into this problem (lack of devs with Macs on the team) periodically.

For sure. Let me know if there's something you'd like me to try.

[CharliePoole]

If you are able to build a test case that excercises the engine at a lower level, that would help. See our Tokenizer tests and TestSelectionParser tests for examples. This might be simpler for you than building the entire engine and runner, although that's an option as well.

Have you looked at the console output yet? It may actually show how the runner is interpreting your where filter.

[OsirisTerje]

It's harder for us if the failure only occurs on a Mac,

I tested it on windows, same results.
And I tested it using dotnet test which invokes the engine.

[livebe01]

It's harder for us if the failure only occurs on a Mac,

I tested it on windows, same results. And I tested it using dotnet test which invokes the engine.

I wanted to add, I get the same error whether running dotnet test or nunit3-console if that provides any additional information...

[CharliePoole]

It's harder for us if the failure only occurs on a Mac,

I tested it on windows, same results. And I tested it using dotnet test which invokes the engine.

OK, I missed that. I labeled it a bug and if nobody gets to it before the 25th, I'll work it.

[stevenaw]

I've been able to try a few things tonight here.
I was able to avoid the parser error by enclosing the test name in single quotes:

dotnet test -- NUnit.Where="test=='Some.Name.Space.SomeClass.TestSomething(EnumValue,\"ABC\")'"

However there still seems to be an issue with the test selection. I can correctly select a test method which expects an int or an enum but not a string. It does appear there is an existing test for exactly that scenario, though, which passes on the engine side.

I have to stop the investigation for now, but if the engine test proves correct XML generation then I am wondering if perhaps this is a framework issue.

dotnet test -- NUnit.Where="test=='Issue1510.Tests.TestSomethingEnum(Option1)'"
dotnet test -- NUnit.Where="test=='Issue1510.Tests.TestSomethingInt(42)'"
dotnet test -- NUnit.Where="test=='Issue1510.Tests.TestSomethingString(\"ABC\")'"

[TestCase(42)]
public void TestSomethingInt(int expected)
{
}

[TestCase(ThisEnum.Option1)]
public void TestSomethingEnum(ThisEnum expected)
{
}

    [TestCase("ABC")]
    public void TestSomethingString(string expected)
    {
        // this one can't be run
    }

[stevenaw]

Free time is a little unpredictable at the moment for me so I've put up a PR with some of the tests I'd written as part of my investigation tonight: #1513