Set securityContext for operator and spilo #341

eberlep · 2022-03-16T18:08:49Z

No description provided.

This reverts commit 381d772.

This reverts commit be4bc1c.

eberlep · 2022-05-12T18:34:37Z

diff RunAsRoot-before.yaml RunAsRoot-after.yaml (k get po pgfits-defaultparams9365a4d-0 -o yaml)

133c133
<       allowPrivilegeEscalation: true
---
>       allowPrivilegeEscalation: false
255c255,257
<   securityContext: {}
---
>   securityContext:
>     runAsGroup: 101
>     runAsUser: 101

Config Flag in Helm Chart working as well

eberlep · 2022-05-12T18:36:26Z

However:

Authentication failure messages in the log, every 10min (su somewhere?)
The backup cronjob does not seem to run automatically, but runs fine when started manually (cron not running?)

eberlep · 2022-05-13T17:24:30Z

From the documentation:

spilo_allow_privilege_escalation Controls whether a process can gain more privileges than its parent process. Required by cron which needs setuid. Without this parameter, certification rotation & backups will not be done. The default is true.

eberlep · 2022-05-13T18:35:47Z

Well, cron indeed seems to be working, but the backups end up in s3://mybucket/$(SCOPE)/, which is probably not what we want.

eberlep · 2022-08-09T12:04:37Z

Spilo container is running as user postgres:

 $ k exec -c postgres -ti pgfits-philippsidecarsd3693-0 -- bash
postgres@pgfits-philippsidecarsd3693-0:~$ whoami                                                                                                                                                                                                                                                                                                                            
postgres

Cron seemts to work as well:

postgres@pgfits-philippsidecarsd3693-0:~$ crontab -l                                                                                                                                                                                                                                                                                                                        
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/14/bin                                                                                                                                                                                                                                                                                
*/15 * * * * envdir "/run/etc/wal-e.d/env" /scripts/postgres_backup.sh "/home/postgres/pgdata/pgroot/data"

postgres@pgfits-philippsidecarsd3693-0:~$ date                                                                                                                                                                                                                                                                                                                              
Tue Aug  9 11:56:57 UTC 2022                                                                                                                                                                                                                                                                  
postgres@pgfits-philippsidecarsd3693-0:~$ envdir "/run/etc/wal-e.d/env" wal-g backup-list                                                                                                                                                                                                                                                                                   
name                          modified             wal_segment_backup_start                                                                                                                                                                                                                                                                                                 
base_000000010000000000000002 2022-08-08T13:15:10Z 000000010000000000000002                                                                                                                                                                                                                                                                                                 
base_000000010000000000000004 2022-08-08T13:15:36Z 000000010000000000000004                                                                                                                                                                                                                                                                                                 
base_000000020000000000000007 2022-08-09T11:33:13Z 000000020000000000000007                                                                                                                                                                                                                                                                                                 
base_000000020000000000000009 2022-08-09T11:45:06Z 000000020000000000000009                                                                                                                                                                                                                                                                                                 
base_00000003000000000000000C 2022-08-09T11:50:12Z 00000003000000000000000C

and after a couple of minutes, one more backup:

postgres@pgfits-philippsidecarsd3693-0:~$ date
Tue Aug  9 12:00:38 UTC 2022
postgres@pgfits-philippsidecarsd3693-0:~$ envdir "/run/etc/wal-e.d/env" wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000002 2022-08-08T13:15:10Z 000000010000000000000002
base_000000010000000000000004 2022-08-08T13:15:36Z 000000010000000000000004
base_000000020000000000000007 2022-08-09T11:33:13Z 000000020000000000000007
base_000000020000000000000009 2022-08-09T11:45:06Z 000000020000000000000009
base_00000003000000000000000C 2022-08-09T11:50:12Z 00000003000000000000000C
base_00000003000000000000000E 2022-08-09T12:00:06Z 00000003000000000000000E

And, just for good measure, backups are stored to the correct bucket as well:

 ~ $ mc ls pgaas-dev/backup-5d1171b6/pgfits-philippsidecarsd3693/basebackups_005/
[2022-08-08 15:15:10 CEST]   400B base_000000010000000000000002_backup_stop_sentinel.json
[2022-08-08 15:15:36 CEST]   400B base_000000010000000000000004_backup_stop_sentinel.json
[2022-08-09 13:33:13 CEST]   402B base_000000020000000000000007_backup_stop_sentinel.json
[2022-08-09 13:45:06 CEST]   402B base_000000020000000000000009_backup_stop_sentinel.json
[2022-08-09 13:50:12 CEST]   402B base_00000003000000000000000C_backup_stop_sentinel.json
[2022-08-09 14:00:06 CEST]   402B base_00000003000000000000000E_backup_stop_sentinel.json
[2022-08-09 14:07:28 CEST]     0B base_000000010000000000000002/
[2022-08-09 14:07:28 CEST]     0B base_000000010000000000000004/
[2022-08-09 14:07:28 CEST]     0B base_000000020000000000000007/
[2022-08-09 14:07:28 CEST]     0B base_000000020000000000000009/
[2022-08-09 14:07:28 CEST]     0B base_00000003000000000000000C/
[2022-08-09 14:07:28 CEST]     0B base_00000003000000000000000E/

UPDATE: And one more...

postgres@pgfits-philippsidecarsd3693-0:~$ date
Tue Aug  9 12:15:26 UTC 2022
postgres@pgfits-philippsidecarsd3693-0:~$ envdir "/run/etc/wal-e.d/env" wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000002 2022-08-08T13:15:10Z 000000010000000000000002
base_000000010000000000000004 2022-08-08T13:15:36Z 000000010000000000000004
base_000000020000000000000007 2022-08-09T11:33:13Z 000000020000000000000007
base_000000020000000000000009 2022-08-09T11:45:06Z 000000020000000000000009
base_00000003000000000000000C 2022-08-09T11:50:12Z 00000003000000000000000C
base_00000003000000000000000E 2022-08-09T12:00:06Z 00000003000000000000000E
base_000000030000000000000010 2022-08-09T12:15:07Z 000000030000000000000010

eberlep · 2022-08-09T12:11:15Z

Prometheus Exporter is still working as well:

 $ k exec -c postgres -ti pgfits-philippsidecarsd3693-0 -- curl http://localhost:9187/metrics | grep pg_ | tail
pg_statio_user_tables_toast_blks_read{datname="postgres",postgres_partition="fel-wps101",relname="postgres_log",schemaname="public",server="pgfits-philippsidecarsd3693-0"} 0
# HELP pg_up Whether the last scrape of metrics from PostgreSQL was able to connect to the server (1 for yes, 0 for no).
# TYPE pg_up gauge
pg_up{postgres_partition="fel-wps101",server="pgfits-philippsidecarsd3693-0"} 1
# HELP pg_wal_activity_last_5_min_size_bytes Current size in bytes of the last 5 minutes of WAL generation. Includes recycled WALs.
# TYPE pg_wal_activity_last_5_min_size_bytes gauge
pg_wal_activity_last_5_min_size_bytes{postgres_partition="fel-wps101",server="pgfits-philippsidecarsd3693-0"} 0
# HELP pg_wal_activity_total_size_bytes Current size in bytes of the WAL directory
# TYPE pg_wal_activity_total_size_bytes gauge
pg_wal_activity_total_size_bytes{postgres_partition="fel-wps101",server="pgfits-philippsidecarsd3693-0"} 2.18104294e+08

eberlep · 2022-08-09T12:12:28Z

And so is fluentbit:

 $ k logs -c postgres-fluentbit pgfits-philippsidecarsd3693-0 | tail
[2] psqlcsv: [1660047068.551345674, {"log"=>"2022-08-09 12:11:08.551 UTC,,,986,"[local]",62f24edc.3da,1,"",2022-08-09 12:11:08 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized""}]
[3] psqlcsv: [1660047068.552138960, {"log"=>"2022-08-09 12:11:08.551 UTC,"postgres","postgres",986,"[local]",62f24edc.3da,2,"authentication",2022-08-09 12:11:08 UTC,9/203,0,LOG,00000,"connection authorized: user=postgres database=postgres application_name=pgq ticker",,,,,,,,,"","client backend""}]
[4] psqlcsv: [1660047068.552272298, {"log"=>"2022-08-09 12:11:08.552 UTC,,,987,"[local]",62f24edc.3db,1,"",2022-08-09 12:11:08 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized""}]
[5] psqlcsv: [1660047068.553109869, {"log"=>"2022-08-09 12:11:08.552 UTC,"postgres","fitsprepdb01",987,"[local]",62f24edc.3db,2,"authentication",2022-08-09 12:11:08 UTC,10/36,0,LOG,00000,"connection authorized: user=postgres database=fitsprepdb01 application_name=pgq ticker",,,,,,,,,"","client backend""}]
[6] psqlcsv: [1660047068.553113449, {"log"=>"2022-08-09 12:11:08.553 UTC,"postgres","postgres",985,"[local]",62f24edc.3d9,3,"idle",2022-08-09 12:11:08 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.003 user=postgres database=postgres host=[local]",,,,,,,,,"pgq ticker","client backend""}]
[7] psqlcsv: [1660047068.553342257, {"log"=>"2022-08-09 12:11:08.553 UTC,,,988,"[local]",62f24edc.3dc,1,"",2022-08-09 12:11:08 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized""}]
[8] psqlcsv: [1660047068.553658286, {"log"=>"2022-08-09 12:11:08.553 UTC,"postgres","postgres",986,"[local]",62f24edc.3da,3,"idle",2022-08-09 12:11:08 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]",,,,,,,,,"pgq ticker","client backend""}]
[9] psqlcsv: [1660047068.553963736, {"log"=>"2022-08-09 12:11:08.553 UTC,"postgres","fitsdb01",988,"[local]",62f24edc.3dc,2,"authentication",2022-08-09 12:11:08 UTC,8/502,0,LOG,00000,"connection authorized: user=postgres database=fitsdb01 application_name=pgq ticker",,,,,,,,,"","client backend""}]
[10] psqlcsv: [1660047068.554898486, {"log"=>"2022-08-09 12:11:08.554 UTC,"postgres","fitsprepdb01",987,"[local]",62f24edc.3db,3,"idle",2022-08-09 12:11:08 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.003 user=postgres database=fitsprepdb01 host=[local]",,,,,,,,,"pgq ticker","client backend""}]
[11] psqlcsv: [1660047068.555811987, {"log"=>"2022-08-09 12:11:08.555 UTC,"postgres","fitsdb01",988,"[local]",62f24edc.3dc,3,"idle",2022-08-09 12:11:08 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=postgres database=fitsdb01 host=[local]",,,,,,,,,"pgq ticker","client backend""}]

eberlep · 2022-08-09T12:20:49Z

Looking at the spec, it looks allright too:

$ k get sts pgfits-philippsidecarsd3693 -o yaml | k neat | yq eval .spec.template.spec -:

containers:
  - name: postgres
    # [..]
    securityContext:
      allowPrivilegeEscalation: true # required for cron
      privileged: false
      readOnlyRootFilesystem: false
# [..]
securityContext:
  runAsGroup: 101
  runAsUser: 101

$ k get deploy postgres-operator -o yaml | k neat | yq eval .spec.template -

spec:
  containers:
    - name: postgres-operator
      # [..]
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1000
      # [..]

…builder image)

Set securityContext for operator and spilo

13b06e9

eberlep linked an issue Mar 16, 2022 that may be closed by this pull request

Check If securityContext can be set for Postgres Pods #326

Open

eberlep added 10 commits March 17, 2022 08:27

Unexport variables

181860d

Merge remote-tracking branch 'origin/main' into securityContext

2cc4782

Fix kubebuilder url

e4f2557

Add debug output

be4bc1c

Debug output for make

381d772

Remove debug output

20e1673

go install instead of go get

62e4276

Revert "Debug output for make"

934b5e5

This reverts commit 381d772.

Revert "Add debug output"

600e03e

This reverts commit be4bc1c.

Make runAsNonRoot configurable

fec15f4

eberlep linked an issue May 12, 2022 that may be closed by this pull request

check parameters #327

Open

eberlep requested a review from majst01 May 12, 2022 18:37

Merge remote-tracking branch 'origin/main' into securityContext

1002022

Try runAsNonRoot, but with privilege escalation (for cron)

28cf662

eberlep mentioned this pull request Jun 15, 2022

Backups are sometimes written to $(SCOPE) instead of <CLUSTERNAME> #378

Closed

eberlep added 3 commits June 28, 2022 08:44

Setting to true instead of not setting it and hoping for the default

51f0b3e

Merge remote-tracking branch 'origin' into securityContext

f0d020e

Merge remote-tracking branch 'origin/main' into securityContext

2cee9fe

eberlep added 2 commits August 9, 2022 14:42

Make RunAsNonRoot revertable

a0376a6

Make RunAsNonRoot revertable

6837490

eberlep added 5 commits August 9, 2022 15:01

Pin minor go version for PRs (aka tests, image itself is build via a …

b4f6d02

…builder image)

Even better, get the go version from the go.mod file

9392e85

Update setup-go

1d9196a

Update linter to prevent it from fucking up the go build env

4ccd9b9

Merge remote-tracking branch 'origin/main' into securityContext

68e34d6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set securityContext for operator and spilo #341

Set securityContext for operator and spilo #341

eberlep commented Mar 16, 2022

eberlep commented May 12, 2022

eberlep commented May 12, 2022

eberlep commented May 13, 2022

eberlep commented May 13, 2022

eberlep commented Aug 9, 2022 •

edited

Loading

eberlep commented Aug 9, 2022

eberlep commented Aug 9, 2022

eberlep commented Aug 9, 2022 •

edited

Loading

Set securityContext for operator and spilo #341

Are you sure you want to change the base?

Set securityContext for operator and spilo #341

Conversation

eberlep commented Mar 16, 2022

eberlep commented May 12, 2022

eberlep commented May 12, 2022

eberlep commented May 13, 2022

eberlep commented May 13, 2022

eberlep commented Aug 9, 2022 • edited Loading

eberlep commented Aug 9, 2022

eberlep commented Aug 9, 2022

eberlep commented Aug 9, 2022 • edited Loading

eberlep commented Aug 9, 2022 •

edited

Loading

eberlep commented Aug 9, 2022 •

edited

Loading