Possible bug in twister gtest harness

[GeekOffTheStreet]

I tried using Twister with the gtest harness and ran into issues with it detecting unit test execution. The behavior for how it parses sample.yaml/testcase.yaml doesn't match what I would expect. I should not that I'm not using pigweed. Consider this simple test case:

#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include <stdio.h>
using namespace testing;

class HelloWorld : public Test {
public:
    HelloWorld() {}
    void SetUp() {}
    void TearDown() {}
};

TEST_F(HelloWorld, foo) {
    EXPECT_TRUE(true);
}

I build that for qemu using:

$ west build -p always -b qemu_leon3--build-dir zephyr-build .
$ /opt/zephyrproject/zephyr/scripts/twister  -vvv -p qemu_leon3 -T ./

If I use the console harness with a sample.yaml resembling:

sample:
  description: Hello World sample, the simplest Zephyr
  name: test hello world
common:
  tags: introduction
  platform_allow: qemu_leon3
  harness: console
  harness_config:
    type: multi_line
    ordered: false
    regex:
      - ".*OK\\s+\\]\\s+HelloWorld.foo.+"

tests:
  HelloWorld.foo:
    tags: introduction

The test works as expected (sample output):

DEBUG   - Determine test cases for test instance: qemu_leon3/HelloWorld.foo
DEBUG   - Determine test cases for test suite: HelloWorld.foo
DEBUG   - Test instance qemu_leon3/HelloWorld.foo already has 1 cases.
DEBUG   - run test: qemu_leon3/HelloWorld.foo
DEBUG   - Reset instance status from 'passed' to None before run.
DEBUG   - Spawning QEMUHandler Thread for qemu_leon3/HelloWorld.foo
DEBUG   - Running qemu_leon3/HelloWorld.foo (qemu)
DEBUG   - Spawning QEMUHandler Thread for qemu_leon3/HelloWorld.foo
DEBUG   - QEMU (232776): [==========] Running 1 test from 1 test case.
DEBUG   - QEMU (232776): [----------] Global test environment set-up.
DEBUG   - QEMU (232776): [----------] 1 test from HelloWorld
DEBUG   - QEMU (232776): [ RUN      ] HelloWorld.foo
DEBUG   - QEMU (232776): [       OK ] HelloWorld.foo (0 ms)
DEBUG   - HARNESS:Console:EXPECTED(1/1):'.*OK\s+\]\s+HelloWorld.foo.+'
DEBUG   - QEMU (232776): [----------] 1 test from HelloWorld (0 ms total)
DEBUG   - QEMU (232776): 
DEBUG   - QEMU (232776): [----------] Global test environment tear-down
DEBUG   - QEMU (232776): [==========] 1 test from 1 test case ran. (10 ms total)
DEBUG   - QEMU (232776): [  PASSED  ] 1 test.
DEBUG   - QEMU (232776): *** Booting Zephyr OS build zephyr-v3.5.0-2466-g654a74ecb7d6 ***
DEBUG   - QEMU (232776) complete (passed) after 2.050739288330078 seconds
DEBUG   - No timeout, return code from QEMU (None): 0
DEBUG   - return code from QEMU (None): 0
DEBUG   - run status: qemu_leon3/HelloWorld.foo passed
INFO    - 1/1 qemu_leon3                HelloWorld.foo                                     PASSED (qemu 2.051s)

INFO    - 1 test scenarios (1 test instances) selected, 0 configurations skipped (0 by static filter, 0 at runtime).

If I try to use the gtest harness:

sample:
  description: Hello World sample, the simplest Zephyr
  name: test hello world
common:
  tags: introduction
  platform_allow: qemu_leon3
  harness: gtest
tests:
  HelloWorld.foo:
    tags: introduction

The runner tries to match HelloWorld.foo.HelloWorld.foo instead of just HelloWorld.foo and fails:

DEBUG   - Determine test cases for test instance: qemu_leon3/HelloWorld.foo
DEBUG   - Determine test cases for test suite: HelloWorld.foo
DEBUG   - Test instance qemu_leon3/HelloWorld.foo already has 1 cases.
DEBUG   - run test: qemu_leon3/HelloWorld.foo
DEBUG   - Reset instance status from 'passed' to None before run.
DEBUG   - Spawning QEMUHandler Thread for qemu_leon3/HelloWorld.foo
DEBUG   - Running qemu_leon3/HelloWorld.foo (qemu)
DEBUG   - Spawning QEMUHandler Thread for qemu_leon3/HelloWorld.foo
DEBUG   - QEMU (234555): [==========] Running 1 test from 1 test case.
DEBUG   - QEMU (234555): [----------] Global test environment set-up.
DEBUG   - QEMU (234555): [----------] 1 test from HelloWorld
DEBUG   - QEMU (234555): [ RUN      ] HelloWorld.foo
DEBUG   - Could not find a matching testcase for HelloWorld.foo.HelloWorld.foo
DEBUG   - QEMU (234555): [       OK ] HelloWorld.foo (0 ms)
Exception in thread qemu_leon3/HelloWorld.foo:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/zephyrproject/zephyr/scripts/pylib/twister/twisterlib/handlers.py", line 902, in _thread
    harness.handle(line)
  File "/opt/zephyrproject/zephyr/scripts/pylib/twister/twisterlib/harness.py", line 573, in handle
    tc is not None and tc == self.tc
AssertionError: gTest error, mismatched tests. Expected HelloWorld.foo.HelloWorld.foo but got None
DEBUG   - return code from QEMU (None): 2
DEBUG   - run status: qemu_leon3/HelloWorld.foo failed
INFO    - 1/1 qemu_leon3                HelloWorld.foo                                      FAILED Timeout (qemu)

I didn't see an obvious way to alter the matching behavior.

@yperess

[Captnlink]

Hi @GeekOffTheStreet,
I get the same thing as you.
Did you find a solution or you reverted back to the regex workaround you made?

[awojasinski]

@Captnlink the code that causes the issue is here

zephyr/scripts/pylib/twister/twisterlib/harness.py

Line 599 in 23fb3b7

class Gtest(Harness):

the GTest class that inherits from Harness class parses output in a strange way. It should be aligned. Also the way test cases are used for Gtest harness it odd. A testcase.yaml file should be used for different compilation types not to describe test cases that are in the source code.