Services failing to start with a state of CrashLoopBackOff

[henrybell]

Describe the bug

The following services are failing to start with a state of CrashLoopBackOff:

• contacts
• userservice
• frontend

To Reproduce

Build the images for the above services and deploy to GKE via skaffold.

Logs

- bank-of-anthos-staging:pod/contacts-77f6b5d6f4-w8kh7: container contacts is backing off waiting to restart
      > [contacts-77f6b5d6f4-w8kh7 contacts] Traceback (most recent call last):
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/bin/gunicorn", line 8, in <module>
      > [contacts-77f6b5d6f4-w8kh7 contacts]     sys.exit(run())
      > [contacts-77f6b5d6f4-w8kh7 contacts]              ^^^^^
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
      > [contacts-77f6b5d6f4-w8kh7 contacts]     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]", prog=prog).run()
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/app/base.py", line 236, in run
      > [contacts-77f6b5d6f4-w8kh7 contacts]     super().run()
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/app/base.py", line 72, in run
      > [contacts-77f6b5d6f4-w8kh7 contacts]     Arbiter(self).run()
      > [contacts-77f6b5d6f4-w8kh7 contacts]     ^^^^^^^^^^^^^
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 58, in __init__
      > [contacts-77f6b5d6f4-w8kh7 contacts]     self.setup(app)
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 93, in setup
      > [contacts-77f6b5d6f4-w8kh7 contacts]     self.log = self.cfg.logger_class(app.cfg)
      > [contacts-77f6b5d6f4-w8kh7 contacts]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/glogging.py", line 194, in __init__
      > [contacts-77f6b5d6f4-w8kh7 contacts]     self.setup(cfg)
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/site-packages/gunicorn/glogging.py", line 262, in setup
      > [contacts-77f6b5d6f4-w8kh7 contacts]     fileConfig(cfg.logconfig, defaults=defaults,
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/logging/config.py", line 83, in fileConfig
      > [contacts-77f6b5d6f4-w8kh7 contacts]     formatters = _create_formatters(cp)
      > [contacts-77f6b5d6f4-w8kh7 contacts]                  ^^^^^^^^^^^^^^^^^^^^^^
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/logging/config.py", line 116, in _create_formatters
      > [contacts-77f6b5d6f4-w8kh7 contacts]     flist = cp["formatters"]["keys"]
      > [contacts-77f6b5d6f4-w8kh7 contacts]             ~~^^^^^^^^^^^^^^
      > [contacts-77f6b5d6f4-w8kh7 contacts]   File "/usr/local/lib/python3.12/configparser.py", line 941, in __getitem__
      > [contacts-77f6b5d6f4-w8kh7 contacts]     raise KeyError(key)
      > [contacts-77f6b5d6f4-w8kh7 contacts] KeyError: 'formatters'

This is caused by a lack of permissions for non-root users on the application and configuration files in /app, resulting in the above error (for logging.conf being inacessible), as the service containers run as non-root users. I have a fix for this incoming.

Related

#517, #2167

[bourgeoisor]

Weird, I (as well as the automated e2e deploys that builds at each PR) cannot reproduce this.

Out of curiosity... Would you mind doing a ls -lash in your local environment inside where you cloned microservices-demo? What sort of machine are you running this into?

[henrybell]

I cloned the repo onto my cloudtop -- I'd prefer not to post these user/group permissions details publicy, so will email directly, thanks!