err: Unable to Get the chaosengine

[niebomin]

My environment is pretty simple. I have an azure k8s cluster, and followed this guide https://istio.io/latest/docs/setup/getting-started/ to setup my environment. In other words, the namespace is enabled with istio injection.

Litmus is setup, and I was trying to run pod-delete experiment. Chaos runner pod is created, but I saw errors though. "pod-delete" pod is also created, but the target pod was not deleted. When I was looking at logs by k logs -f pod-delete-4y48y4-ql64c -c pod-delete-4y48y4, I saw error

Unable to initialise probes details from chaosengine, err: Unable to Get the chaosengine, err: Get \"https://10.0.0.1:443/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosengines/bookinfo-chaos\": dial tcp 10.0.0.1:443: connect: connection refused

This is my engine yaml, https://drive.google.com/file/d/1HAaMLamHS3BZP6SDNtD_-YO1pdl46vnh/view?usp=sharing

Litmus version is 1.9.0

[ksatchit]

cc: @ispeakc0de

[ksatchit]

@niebomin could you please provide the logs for following pods:

• chaos-operator
• chaos-runner
• pod-delete exp

The kubectl describe output of the chaosengine will help too.

[niebomin]

Thanks for the response. The environment was taken down. Will take a while to get it back.

[niebomin]

Command: k logs -f bookinfo-chaos-runner -c chaos-runner
Output:

W1029 09:14:42.550178       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2020-10-29T09:14:42Z" level=info msg="Experiments details are as follows" Service Account Name=pod-delete-sa Engine Namespace=default Experiments List="[pod-delete]" Engine Name=bookinfo-chaos appLabels="app=details" appKind=
time="2020-10-29T09:14:42Z" level=info msg="Getting the ENV Variables"
time="2020-10-29T09:14:43Z" level=info msg="Preparing to run Chaos Experiment: pod-delete"
time="2020-10-29T09:14:43Z" level=info msg="Validating configmaps specified in the ChaosExperiment & ChaosEngine"
time="2020-10-29T09:14:43Z" level=info msg="Validating secrets specified in the ChaosExperiment & ChaosEngine"
time="2020-10-29T09:14:44Z" level=info msg="Validating HostFileVolumes details specified in the ChaosExperiment"
time="2020-10-29T09:14:54Z" level=info msg="Started Chaos Experiment Name: pod-delete, with Job Name: pod-delete-lddg4u"

pod-delete-lddg4u-wmtmm status is Error

Command: k logs -f pod-delete-lddg4u-wmtmm

Output:

error: a container name must be specified for pod pod-delete-lddg4u-wmtmm, choose one of: [pod-delete-lddg4u istio-proxy] or one of the init containers: [istio-init]
bnie@bnie-a03 k8-pod-delete % k logs -f pod-delete-lddg4u-wmtmm -c pod-delete-lddg4u
W1029 09:14:54.209448       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2020-10-29T09:14:54Z" level=info msg="Experiment Name: pod-delete"
time="2020-10-29T09:14:54Z" level=info msg="[PreReq]: Getting the ENV for the  experiment"
time="2020-10-29T09:14:54Z" level=fatal msg="Unable to initialise probes details from chaosengine, err: Unable to Get the chaosengine, err: Get \"https://10.0.0.1:443/apis/litmuschaos.io/v1alpha1/namespaces/default/chaosengines/bookinfo-chaos\": dial tcp 10.0.0.1:443: connect: connection refused"

Operator log: https://drive.google.com/file/d/1EeoOg_fTOtdDuhdk4-DmAysdwEu_a55G/view?usp=sharing. Please look at the last part

Engine describe: https://drive.google.com/file/d/1Rnmmk3KrePMkc5-Nq5gO36Fi0ResckMx/view?usp=sharing

[niebomin]

I kind of think it's because the pod-delete container whose image is litmuschaos/go-runner:1.9.0. After I switched it back to 1.7, it worked.

[ksatchit]

This is interesting. @ispeakc0de do you have a comment on this? The error is in being able to access chaosengine object by the experiment pod - until then the operator and runner are successfully able to generate events against it. @niebomin was the error a one off or do you see that everytime you run the experiment on 1.9.0? Also, just to confirm were you using the chaosexperiment CR also specific to 1.9.0? (https://hub.litmuschaos.io/api/chaos/1.9.0?file=charts/generic/pod-delete/experiment.yaml)

[niebomin]

I can see it every time. I downloaded the source, and was using the yaml file locally.