in-cluster-cp4waiops-aimanager ArgoCD App Status Stuck Progressing

[nmm131]

Hello,

I am configuring IBM Cloud Pak for Watson AIOps (CP4WAIOPS) all-in-one using a custom build and the following pods are stuck in a Pending state:

oc get pods --field-selector 'status.phase=Pending' -A

NAMESPACE               NAME                                             READY   STATUS                       RESTARTS   AGE
cp4waiops               aiops-installation-edb-postgres-1-initdb-fjfhr   0/1     ImagePullBackOff             0          24m
cp4waiops               c-example-redis-m-0                              0/4     Init:ImagePullBackOff        0          16m
cp4waiops               c-example-redis-m-1                              0/4     Init:ImagePullBackOff        0          16m
cp4waiops               c-example-redis-m-2                              0/4     Init:ImagePullBackOff        0          16m
cp4waiops               cp4waiops-connectors-deploy-8655f884d6-wx4b9     0/1     Init:ImagePullBackOff        0          7m54s
cp4waiops               gateway-kong-99446b75c-n6vlk                     0/2     ImagePullBackOff             0          19m
cp4waiops               gateway-kong-config-svc-27886670-br85n           0/1     ImagePullBackOff             0          14m
cp4waiops               gateway-kong-config-svc-27886680-kgbk2           0/1     ImagePullBackOff             0          4m5s
cp4waiops               gateway-kong-post-install-resources-6ghc2        0/1     ImagePullBackOff             0          18m
cp4waiops               sre-tunnel-controller-0                          0/1     ContainerCreating            0          21m
ibm-common-services     common-web-ui-7b7bd46998-vkkms                   0/1     CreateContainerConfigError   0          27m
ibm-common-services     create-postgres-license-config-25hk5             0/1     Init:ImagePullBackOff        0          27m
ibm-common-services     icp-mongodb-0                                    0/2     Init:0/2                     0          17m
noi                     evtmanager-ea-noi-layer-easetupomnibus-xbqlc     0/1     Init:ImagePullBackOff        0          30m
noi                     evtmanager-zookeeper-0                           0/1     Init:ImagePullBackOff        0          29m
openshift-marketplace   ibm-operator-catalog-bzwqn                       0/1     ContainerCreating            0          5s

Here are my infrastructure and configuration details:

CP4WAIOPS version: 3.6
OpenShift version: 4.10
Default StorageClass: ibmc-file-gold-gid
All-in-one Helm Parameters:
- cp4waiops.storageClass: ibmc-file-gold-gid
- cp4waiops.storageClassLargeBlock: ibmc-block-gold

Currently, I have completed the following steps:

• installed OpenShift GitOps
• updated ArgoCD SSL certificates
• granted ArgoCD cluster-admin permissions
• configured ArgoCD for OpenShift
• set the default StorageClass
• updated the global image pull secret
• created an all-in-one (with parameters to configure IBM Cloud storage) ArgoCD application

I confirmed that I am able to login to the cp.icr.io registry using the Entitlement Key that I used to update the global image pull secret:

docker login cp.icr.io --username cp

Password: 
Login Succeeded

The global image pull secret (named pull-secret in the openshift-config namespace) is configured as follows:

Registry server address: cp.icr.io
Username: cp
Password: <ENTITLEMENT_KEY_HERE>

I also created a secret in the cp4waiops namespace with the following configuration, which did not solve my issue:

Name: ibm-entitlement-key
Registry server address: cp.icr.io
Username: cp
Password: <ENTITLEMENT_KEY_HERE>

I also reloaded my cluster's nodes after both pull-secret and ibm-entitlement-key secrets existed with the Entitlement Key value, which did not solve my issue.

All ArgoCD apps are Healthy and Synced except for the in-cluster-cp4waiops-aimanager application, which is stuck Progressing and Synced. I noticed all ArgoCD applications' Target Revision is release-3.6 except for the following ArgoCD apps:

• argocd-custom
• cp-shared
• in-cluster-cp4waiops-aimanager
• in-cluster-cp4waiops-eventmanager

I confirmed that I specified a Revision of release-3.6 when creating my config/argocd/openshift and config/all-in-one Argo CD apps.
I have repeated these steps a few times and the results are the same.