Create trust bundle based on Debian bookworm

[SgtCoDFish]

We currently provide cert-manager-package-debian which contains the trust store from Debian bullseye. That trust store hasn't been updated for a while - since it hasn't needed to be - but it won't be updated to add new CA certificates or to prune CA certificates which have expired.

Since there's a new stable version of Debian - Debian bookworm - we could provide a new trust bundle which uses the ca-certificates package from that new version. That would be more "up to date" for users who want that.

We can create a new bundle for bookworm and update to use it by default in trust-manager. This would involve copying the "debian" bundle here into a new folder ("debian-bookworm") and then updating the new bundle to use bookworm rather than bullseye.

There's some GCB CI/CD for our trust bundles, which is manually configured in the cert-manager GCP org, but I can set that up. The GCB file here would need to be copied and the new version would need to be updated to bookworm.

ℹ️ This would be a great issue for an open source community member to tackle!

[arsenalzp]

Hello,
I can pick it up, but I would like to know - is there any terms for this issue?

[SgtCoDFish]

I can pick it up, but I would like to know - is there any terms for this issue?

Sorry, I didn't see this until now!

Could you explain what you mean by "terms" there?

[arsenalzp]

I can pick it up, but I would like to know - is there any terms for this issue?

Sorry, I didn't see this until now!

Could you explain what you mean by "terms" there?

I meant "deadlines".

[SgtCoDFish]

Ah, no deadlines! We can be flexible, no rush 😁

[arsenalzp]

Could you please assign it to me?

[erikgb]

/assign @arsenalzp

I think you should be able to self-assign:

Ref. https://github.com/jpmcb/prow-github-actions/blob/main/docs/commands.md

[arsenalzp]

Hello,
I'm back from vacation and ready to work.
Could you be so kind to explain me why we need to copy old trust package stuff (in debian folder) instead just change base image directive to use the latest Debian docker package?

[SgtCoDFish]

Could you be so kind to explain me why we need to copy old trust package stuff (in debian folder) instead just change base image directive to use the latest Debian docker package?

Sure, I'll try to explain my thinking!

Basically, the list of CAs trusted by each version of Debian will vary. Bookworm trusts new certs that bullseye doesn't.

I'd prefer for people to be able to make the choice of what they trust by themselves. It's OK for us to change the default in trust-manager (and there has to be a default), but I'd like to give people who already use the bullseye images to continue using those.

Plus if people are going to keep on using bullseye then we need a build pipeline for updating it in the future!

Does that make sense?

[arsenalzp]

Yes, sure it is clear for me.
But the issue is that Debian trusted bundle sets up during the building/deployment process by using Init containers features.
Any suggestion/wishes how to allow end users to define bundle version?
P.S.
We should change help charts as well, there is 'hard-coded' version

# -- Tag for the default package image
  tag: "20210119.0"

[erikgb]

Basically, the list of CAs trusted by each version of Debian will vary. Bookworm trusts new certs that bullseye doesn't.

@SgtCoDFish could you please explain why any user would prefer to trust an outdated trust bundle?

[arsenalzp]

I guess it is for some reason such as the certificate chain of trust which is based on old CA. Some applications/certificates were not updated to use new CA, something like that.

[erikgb]

I guess it is for some reason such as the certificate chain of trust which is based on old CA. Some applications/certificates were not updated to use new CA, something like that.

But the Debian trust roots are based on publicly trusted roots? And I don't think any root certs will be removed before they expire or are revoked. As a user, I would prefer to use an updated public trust bundle.

[arsenalzp]

I guess it is for some reason such as the certificate chain of trust which is based on old CA. Some applications/certificates were not updated to use new CA, something like that.

But the Debian trust roots are based on publicly trusted roots? And I don't think any root certs will be removed before they expire or are revoked. As a user, I would prefer to use an updated public trust bundle.

You are right but some CAs might be revoked so it would lead to issue with the new CA bundle. I guess that's why @SgtCoDFish created this issue.
Of course we should indulge users to use outdated CA bundles, however we shouldn't break users applications as well.
Anyway, decision is fully up to you, colleagues! :-)

[SgtCoDFish]

could you please explain why any user would prefer to trust an outdated trust bundle?

I think that they'd probably not use the term "outdated" - Debian bullseye will be supported for years still.

It's kinda as simple as: if you use the new version, you're potentially trusting more stuff. I just like that people can have the choice, you know?

[arsenalzp]

Hello,
What is the final decision regarding this issue?

[erikgb]

I am not sure if there is a "final decision". 😉 But I will try to summarize my view on this, taking the comments from @SgtCoDFish into account.

Should which defaultCAs trust bundle to use be configurable on the Bundle API level? That will require changes to the API as it is now just a boolean true/false. IMO this is a complexity we don't want/need.

IMO the defaultCAs trust bundle should be configurable when provisioning trust-manager. I also think the easiest solution is to just allow the user to choose image for the init-container to select between the bullseye and bookworm trust bundle. And that we should have CI providing updated tags for both images.

[arsenalzp]

The best and sophisticated solution is to allow user to leverage trusted bundle image with helm chart parameters.
However, @SgtCoDFish has his own point of view on this issue :-)

[SgtCoDFish]

I absolutely agree people should be able to configure this in Helm parameters - which they can today, right?

trust-manager/deploy/charts/trust-manager/values.yaml

Line 33 in ad150e5

defaultPackageImage:

I also think the easiest solution is to just allow the user to choose image for the init-container to select between the bullseye and bookworm trust bundle. And that we should have CI providing updated tags for both images.

This is the way! We can update the default to bookworm, but leave the bullseye image for older versions of trust-manager and for anyone who chooses that.

[arsenalzp]

I realised it.
So the task is just create new docker image by using the latest debian base image, push it to the registry, label it as latest, change Helm chart to use the latest image. Am I right?
Sorry for so many questions, thanks all of you for your patience!

[erikgb]

I don't think we should use latest tags, as they are not reproducible.

[arsenalzp]

I don't think we should use latest tags, as they are not reproducible.

I agree, they don't reflect exact version of image, which is Debian release in our case.
In your code I see a number I guess it reflects some date. Can I use something like that to define distinct Debian release for docker image with CA bundle?

[SgtCoDFish]

In your code I see a number I guess it reflects some date

This is the version number of the package - Debian uses dates for ca-certificates.

$ docker run -it --rm docker.io/library/debian:bookworm-slim bash
$ root@91a2868ce75f:/# apt update && apt install ca-certificates
...
$ apt show ca-certificates
...
Version: 20230311
...

We also add a number on the end so that we can release new versions of the same package (e.g. if we rebuild the go binary).

So I guess for bookworm our first version should look like 20230311.0

[erikgb]

/priority backlog

[cert-manager-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale

[erikgb]

/remove-lifecycle stale

@arsenalzp Are you still interested in looking into this issue?

[arsenalzp]

Yes, let's close this issue.
What was exact decision regarding this problem?
Can just only change defaultPackageImage.tag value in values.yaml?
However, you should push new version of Docker image due to insufficient privileges in jetstack / cert-manager-package-debian repository.

[SgtCoDFish]

What was exact decision regarding this problem?

I don't think we took any final decision! Mostly there just isn't much demand for this - the current bundle isn't massively different to a more "modern" one. I'd still like to provide a more modern one (and move to that) but it's just not the biggest deal!

I'd probably rather have this be a different image, rather than just a new tag - just to avoid any risk of confusion. But I don't think this will get worked on any time soon so that doesn't really matter much!

[pddg]

FYI: Since Go 1.23, it has been changed so that an error occurs when trying to parse a certificate with a negative serial number.
golang/go@db13584

The ca-certificates of bullseye contains a CA certificate of CN=EC-ACC, which has a negative serial number.

$ while openssl x509 -noout -text 2>/dev/null; do :; done < bundle.pem| grep 'Serial Number' -A 3 | grep 'Negative' -A 2 -B 1
        Serial Number:
             (Negative)11:d4:c2:14:2b:de:21:eb:57:9d:53:fb:0c:22:3b:ff
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), OU=Serveis Publics de Certificacio, OU=Vegeu https://www.catcert.net/verarrel (c)03, OU=Jerarquia Entitats de Certificacio Catalanes, CN=EC-ACC

This certificate seems to viloate RFC 5280.
https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.2

The serial number MUST be a positive integer assigned by the CA to
each certificate. It MUST be unique for each certificate issued by a
given CA (i.e., the issuer name and serial number identify a unique
certificate). CAs MUST force the serialNumber to be a non-negative
integer.

（However, the RFC also notes the following）

Note: Non-conforming CAs may issue certificates with serial numbers
that are negative or zero. Certificate users SHOULD be prepared to
gracefully handle such certificates.

This may cause an error when an application built with Go 1.23 or later tries to read the defaultCAs distributed by trust-manager. This actually occurred in my environment and crashed the application.

This can currently be avoided by setting the environment variable GODEBUG=x509negativeserial=1, but this may be removed in future versions.
https://pkg.go.dev/crypto/x509#ParseCertificate

Before Go 1.23, ParseCertificate accepted certificates with negative serial numbers. This behavior can be restored by including "x509negativeserial=1" in the GODEBUG environment variable.

EC-ACC seems to have been removed in bookworm (ca-certificates 20230311).
http://metadata.ftp-master.debian.org/changelogs/main/c/ca-certificates/unstable_changelog
https://packages.debian.org/bookworm/ca-certificates

[logan2211]

FYI: Since Go 1.23, it has been changed so that an error occurs when trying to parse a certificate with a negative serial number. golang/go@db13584

The ca-certificates of bullseye contains a CA certificate of CN=EC-ACC, which has a negative serial number.

I am hitting a problem with ingress-nginx when using trust-manager bundles that have useDefaultCAs: true set.

When using such bundles with ingress-nginx v1.11.4+, the bundle is ignored since ingress-nginx controller v1.11.4+ is built using Go 1.23, which rejects the bundle outright due to the negative serial CA cert present.

Please update the ca-certificates that trust-manager uses to a modern release.

[erikgb]

@logan2211 There is an open PR to fix the existing bundle by filtering out the certificates with negative serial numbers: #515. I expect this PR to be merged early next week and released soon. Stay tuned!