bug: Test case erroneously run when more than one tag in inventory

[kosichd]

When using tags with the anta.tests.interfaces/VerifyInterfacesStatus test, there seems to be an issue where tests are being executed that should not be.

This seems to happen when a switch has more than one tag in the inventory file.

In the below example catalog and inventory files, "management_switch" has two tags, 'osw' and 'DUB'

When running the NRFU command with --tags osw, it should only execute InterfaceStatus against interfaces E51 and E52. However, it seems to be running it against E53/E54 also, which is actually part of the 'DUB' tag, not 'osw'

This causes the test case to fail -

Steps to recreate:

1. Use the below to create inventory and catalog files
2. From the CLI, run:

anta nrfu --username USERNAME--password PASSWORD --inventory recreate_inventory.yml  --catalog recreate_catalog.yml  --tags osw

inventory file

anta_inventory:  
  hosts:
  - host: x.x.x.x
    name: "switch1"
    tags: ['DUB']
  - host :  x.x.x.x
    name: "management_switch"
    tags: ['osw', 'DUB']

catalog file

anta.tests.interfaces:
- VerifyInterfacesStatus:
   filters:
     tags: ['DUB', 'GRN']
   interfaces:
     - name: Ethernet51
       status: up
     - name: Ethernet52
       status: up
     - name: Ethernet53
       status: up
     - name: Ethernet54
       status: up
- VerifyInterfacesStatus:
   filters:
     tags: ['osw']
   interfaces:
     - name: Ethernet51
       status: up
     - name: Ethernet52
       status: up

[gmuloc]

Hello @kosichd this should have be fixed in main in #903
Could you please it run from main and let us know if this fixes your issue?

We hope to release v1.2.0 soonish

[kosichd]

@gmuloc I tried this with the updated code in #903 for runner.py, but still seeing the same behavior

[kosichd]

@gmuloc i also tried with cloning the repo to a new directory to get the most recent v1.1.0 updates, and issue still appears to be there.

[gmuloc]

Hi @kosichd - so v1.1.0 was released before #903 and does indeed have the bug

anta  nrfu -i inventory.yml -c catalog.yml --tags osw
╭────────────────────── Settings ──────────────────────╮
│ - ANTA Inventory contains 2 devices (AsyncEOSDevice) │
│ - Tests catalog contains 2 tests                     │
╰──────────────────────────────────────────────────────╯

[22:40:35] INFO     Preparing ANTA NRFU Run ...                                               tools.py:294
           INFO     Connecting to devices ...                                                 tools.py:294
[22:40:36] INFO     Connecting to devices completed in: 0:00:00.816.                          tools.py:302
           INFO     Preparing the tests ...                                                   tools.py:294
           INFO     Preparing the tests completed in: 0:00:00.004.                            tools.py:302
           INFO     --- ANTA NRFU Run Information ---                                        runner.py:270
                    Number of devices: 2 (1 established)
                    Total number of selected tests: 2
                    Maximum number of open file descriptors for the current ANTA process:
                    16384
                    ---------------------------------
           INFO     Preparing ANTA NRFU Run completed in: 0:00:00.835.                        tools.py:302
           INFO     Running ANTA tests ...                                                    tools.py:294
           INFO     Running ANTA tests completed in: 0:00:00.177.                             tools.py:302
           INFO     Cache statistics for 'management_switch': 1 hits / 2 command(s) (50.00%)  runner.py:75
  • Running NRFU Tests...100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:00 • 0:00:00

                                            All tests results
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃                   ┃                 ┃             ┃                  ┃ Test            ┃               ┃
┃ Device            ┃ Test Name       ┃ Test Status ┃ Message(s)       ┃ description     ┃ Test category ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ management_switch │ VerifyInterfac… │ failure     │ The following    │ Verifies the    │ Interfaces    │
│                   │                 │             │ interface(s) are │ status of the   │               │
│                   │                 │             │ not configured:  │ provided        │               │
│                   │                 │             │ ['Ethernet51',   │ interfaces.     │               │
│                   │                 │             │ 'Ethernet52']    │                 │               │
├───────────────────┼─────────────────┼─────────────┼──────────────────┼─────────────────┼───────────────┤
│ management_switch │ VerifyInterfac… │ failure     │ The following    │ Verifies the    │ Interfaces    │
│                   │                 │             │ interface(s) are │ status of the   │               │
│                   │                 │             │ not configured:  │ provided        │               │
│                   │                 │             │ ['Ethernet51',   │ interfaces.     │               │
│                   │                 │             │ 'Ethernet52',    │                 │               │
│                   │                 │             │ 'Ethernet53',    │                 │               │
│                   │                 │             │ 'Ethernet54']    │                 │               │
└───────────────────┴─────────────────┴─────────────┴──────────────────┴─────────────────┴───────────────┘

However in main currently (so including PR 903) here is the output I get (which is as expected):

anta  nrfu -i inventory.yml -c catalog.yml --tags osw
╭────────────────────── Settings ──────────────────────╮
│ - ANTA Inventory contains 2 devices (AsyncEOSDevice) │
│ - Tests catalog contains 2 tests                     │
╰──────────────────────────────────────────────────────╯

[22:42:17] INFO     Preparing ANTA NRFU Run ...                                                                                                                                                          tools.py:293
           INFO     Connecting to devices ...                                                                                                                                                            tools.py:293
[22:42:18] INFO     Connecting to devices completed in: 0:00:00.805.                                                                                                                                     tools.py:301
           INFO     Preparing the tests ...                                                                                                                                                              tools.py:293
           INFO     Preparing the tests completed in: 0:00:00.014.                                                                                                                                       tools.py:301
           INFO     --- ANTA NRFU Run Information ---                                                                                                                                                   runner.py:287
                    Number of devices: 2 (1 established)
                    Total number of selected tests: 1  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                    Maximum number of open file descriptors for the current ANTA process: 16384
                    ---------------------------------
           INFO     Preparing ANTA NRFU Run completed in: 0:00:00.837.                                                                                                                                   tools.py:301
           INFO     Running ANTA tests ...                                                                                                                                                               tools.py:293
           INFO     Running ANTA tests completed in: 0:00:00.209.                                                                                                                                        tools.py:301
           INFO     Cache statistics for 'management_switch': 0 hits / 1 command(s) (0.00%)                                                                                                              runner.py:78
  • Running NRFU Tests...100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 • 0:00:00 • 0:00:00

                                                                                                All tests results
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Device            ┃ Test Name              ┃ Test Status ┃ Message(s)                  ┃ Test description                                                                                      ┃ Test category ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ management_switch │ VerifyInterfacesStatus │ failure     │ Ethernet51 - Not configured │ Verifies the operational states of specified interfaces to ensure they match expected configurations. │ Interfaces    │
│                   │                        │             │ Ethernet52 - Not configured │                                                                                                       │
└───────────────────┴────────────────────────┴─────────────┴─────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────┘  

I think the next release should fix your issue. How have you tested 903?