[RC2] [Functest K8s] kube_bench_master TC taint issue

[sultetveny]

I've faced with the following issue running the kubernetes security test using kube_bench_master.

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-security:v1.23 /bin/bash

# then within the container
run_tests -t kube_bench_master

Test case failed. For more information please check attached file.
functest-kubernetes.debug (1).log

+-------------------------+----------------------------------------------------------+
|         ENV VAR         |                          VALUE                           |
+-------------------------+----------------------------------------------------------+
|         CI_LOOP         |                          daily                           |
|          DEBUG          |                           true                           |
|     DEPLOY_SCENARIO     |                 k8s-nosdn-nofeature-noha                 |
|      INSTALLER_TYPE     |                         unknown                          |
|        BUILD_TAG        |                                                          |
|        NODE_NAME        |                                                          |
|       TEST_DB_URL       |     http://testresults.opnfv.org/test/api/v1/results     |
|     TEST_DB_EXT_URL     |                                                          |
|     S3_ENDPOINT_URL     |                                                          |
|        S3_DST_URL       |                                                          |
|       HTTP_DST_URL      |                                                          |
+-------------------------+----------------------------------------------------------+

2023-02-24 14:10:38,049 - xtesting.ci.run_tests - INFO - Loading test case 'kube_bench_master'...
2023-02-24 14:10:38,418 - xtesting.ci.run_tests - INFO - Running test case 'kube_bench_master'...

2023-02-24 14:30:38,520 - xtesting.ci.run_tests - ERROR - 

Please fix the testcase kube_bench_master.
All exceptions should be caught by the testcase instead!

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/xtesting/ci/run_tests.py", line 171, in run_test
    test_case.run(**kwargs)
  File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 212, in run
    self.details["report"] = ast.literal_eval(self.pod_log)
  File "/usr/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 0
    
SyntaxError: unexpected EOF while parsing
2023-02-24 14:30:38,522 - xtesting.ci.run_tests - ERROR - The test case 'kube_bench_master' failed.
2023-02-24 14:30:38,522 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_ERROR

The content of the env file successfully read as you see below the env variables set within the container properly. However their values somehow not respected (e.g., NON_BLOCKING_TAINTS).
The K8S cluster contains 3 controllers, 2 workers, 1 edge node.

bash-5.1# printenv
HOSTNAME=947d26a68598
GCR_REPO=gcr-io.repo.cci.nokia.net
PWD=/
container=podman
HOME=/root
KUBE_TEST_REPO_LIST=/home/opnfv/functest/repositories.yml
DEPLOY_SCENARIO=k8s-nosdn-nofeature-noha
TERM=xterm
SHLVL=1
NON_BLOCKING_TAINTS=is_control,is_edge
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG=true
K8S_GCR_REPO=k8s-gcr-io.repo.cci.nokia.net
_=/bin/printenv
bash-5.1# 

kubectl get events -A --watch

NAMESPACE          LAST SEEN   TYPE      REASON             OBJECT                        MESSAGE
kube-bench-f4dqn   56s         Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.
kube-bench-f4dqn   57s         Normal    SuccessfulCreate   job/kube-bench-master         Created pod: kube-bench-master-t5l89
kube-bench-f4dqn   0s          Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.
kube-bench-f4dqn   0s          Warning   FailedScheduling   pod/kube-bench-master-t5l89   0/6 nodes are available: 1 node(s) had taint {is_edge: true}, that the pod didn't tolerate, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {is_control: true}, that the pod didn't tolerate.

kubectl get pods -n kube-bench-f4dqn -o wide

NAME                      READY   STATUS    RESTARTS   AGE     IP       NODE     NOMINATED NODE   READINESS GATES
kube-bench-master-t5l89   0/1     Pending   0          2m53s   <none>   <none>   <none>           <none>

As pointed out in kubernetes project, I should open a ticket about the issue here. kubernetes/kubernetes#116213