Breaking change: .NET 9 is Y2038 compatible (Arm32 glibc); .NET 8 is not

[richlander]

We are making a major change to our Linux Arm32 support that affects all .NET versions.

• .NET 9 will only support Y2038-compatible Linux Arm32 distros, like Ubuntu 24.04 and (we expect) Debian 13.
• .NET 8 will remain Y2038 incompatible on Linux Arm32. Same applies to earlier .NET versions.

From Ubuntu 24.04 release notes:

Ubuntu 24.04 LTS solves the Year 2038 problem that existed on armhf. More than a thousand packages have been updated to handle time using a 64-bit value rather than a 32-bit one, making it possible to handle times up to 292 billion years in the future.

That sounds great. We are changing the way we build .NET 9 for Arm32 to absorb this breaking change.

This change doesn't affect our Arm32 musl build (for Alpine). The change to support Y2038 for musl was made with .NET 6.

.NET 8 will remain Y2038 incompatible on Linux Arm32 and .NET 9 will only support recent distro versions (for Arm32). .NET users will need to carefully select the .NET version they use in their Arm32 deployments.

• .NET 9 -- Y2038 compatible; earliest supported OSes:
  • Ubuntu 22.04
  • Debian 12
• .NET 8 -- Y2038 incompatible; latest supported OS:
  • Ubuntu 22.04
  • Debian 12

An implication of this approach is that we will not publish .NET container images for Ubuntu 24.04 for the Arm32 architecture.

Note: Exact version information is described in the Linux compatibility section of Supported OS Policy documents, for each .NET version.

This problem does not affect Arm64 or x64. .NET 8 and NET 9 are supported on the same Debian and Ubuntu versions for those architectures.

Y2038 incompatible .NET builds running on a Y2038 compatible Arm32 distro may see the following error.

The SSL connection could not be established, see inner exception.
The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

It is possible that there are other error conditions.

Work will be tracked at: dotnet/runtime#101944

Details

We discovered that .NET 8 is not compatible with Arm32 on Ubuntu 24.04 as part of producing new noble container images. In particular, we see a break in OpenSSL API calls (used for HttpClient and HTTPS).

As suggested, there is a solution to this problem, however, it requires that our build targets a glibc version 2.34 that was first made available in Ubuntu 22.04 (in terms of Ubuntu versions). But because we build on a long stable Linux distro to ensure maximum compatibility, using a new version of glibc to leverage the special compiler flags to enable Y2038 time would unfortunately mean that we'd limit the number of systems where .NET 8 could run, and ultimately we decided this was not in the best interest of .NET users.

Y2038 support -- in general -- is a breaking change. As suggested, there is a mitigation for it. Software that targets old glibc versions may be broken. Software that targets new glibc versions and uses the correct compiler flags will work correctly.

We have two choices for .NET Arm32 support:

• Support Y2038 with .NET 9 and leave .NET 8 as Y2038 incompatible.
• Change the .NET 8 Arm32 build to target Ubuntu 22.04 instead of Ubuntu 16.04. This is a breaking change. For example, .NET 8 deployments on Raspberry Pi OS "bullseye" flavor would break (fail to launch).

If we could go back in time, we might make this change with .NET 8. However, since Raspberry Pi OS trails Debian releases and Debian is yet to make this change, the .NET 9/10 pair is probably the optimal timeframe.

We've decided to make .NET 9 (and future releases) Y2038 compatible on Arm32 as the best overall solution.

[raffaeler]

Would it be possible to force .NET9 work on previous versions of the OS using an environment variable?
If yes, what about the opposite (running .NET 8 on v24 and later)?
Thank you

[richlander]

We are not going to add a block, so the environment variable won't be needed.

.NET 9 (for Arm32) will be compiled with a more recent glibc so that TIME_BITS can be used. This is the standard solution to the problem. That will cause .NET 9 to fail to load on Debian 11, for example, with missing glibc exports.

.NET 8 will have the behavior you see in the issue on Y2038 compatible distros. There straightforward fix is to use a newer glibc, but then we'd lose compat with older OSes, which we don't want to do.

In retrospect, this conversation was raised back in January with dotnet/runtime#96460. We could have started acting earlier, although we could have had the same options available so that wouldn't have changed much.

This issue also provides some insight: openssl/openssl#16401.

[jkotas]

Would it be possible to force .NET9 work on previous versions of the OS using an environment variable?

In order to have one Y2038 compatible binary that works on both Y2038 compatible and Y2038 incompatible systems, we would have to create 32-bit time_t and 64-bit time_t clones of all code paths that use time_t and change the calling code to invoke one or the other depending on the time_t size of the underlying system. It would be very complicated and fragile solution. We do not think that it is worth it to spend time on creating this solution. (Controlling the behavior via an environment variable would not help.)

If you need .NET 9 binaries that work on Y2038 incompatible systems, you can build your own.

[raffaeler]

Thank you @richlander and @jkotas .
I understand better now. I hoped the changes mentioned by @jkotas be simpler.
I am only afraid of uncontrolled and difficult to detect failures in embedded/IoT scenarios. I have no problem in keeping updated my own apps, but we all known that a good percentage of apps often do not follow this habit.

[richlander]

The failure modes are pretty obvious and it's only a problem after updating an OS, for example to Debian 12 to 13 (or Raspberry Pi OS derivatives).

Leaving an existing OS and app in production until 2038 will also produce similar failure modes.

Folks using existing Arm32 OSes (who rely on glibc like .NET does) will need to do something.

[jkotas]

Leaving an existing OS and app in production until 2038 will also produce similar failure modes.

It is likely that the existing OS and apps will start exhibiting problems sooner than 2038. The problem is not limited to current date >= 2038. The problem will also occur whenever time_t needs to represent date >= 2038. openssl/openssl#16401 is an example of such situation.

[raffaeler]

Leaving an existing OS and app in production until 2038 will also produce similar failure modes.

It is likely that the existing OS and apps will start exhibiting problems sooner than 2038. The problem is not limited to current date >= 2038. The problem will also occur whenever time_t needs to represent date >= 2038. openssl/openssl#16401 is an example of such situation.

I am not sure the problems are necessarily straightforward to detect, that's my only point.
I agree that you have to do something in any case. I'm wondering whether detecting the bugged glibc and produce a warning on the terminal output can be an option.

[jkotas]

.NET 9 will fail to launch on glibc that is Y2038 incompatible.

[richlander]

There is a discussion about delaying this plan by one release.

Today, the 8.0 container tag we offer on each of our repos is multi-arch, offering arm32, arm64, and amd64 images. 8.0-jammy, and 8.0-alpine are the same. That means that if you pull an 8.0 tag on a 32-bit Raspberry Pi OS, you will get a good image. Excellent.

If we go forward with this plan, the same will not be true for the 9.0 tag, but would be true for 9.0-noble and 9.0-alpine tags. That would be very unfortunate and would require extra documentation and training. Arm32 container users pulling 9.0 tags, even on Ubuntu 24.04, would get an error that a manifest entry was not found.

If we push this plan one release further, then 9.0 will have Arm32 images, but 9.0-noble will not. That outcome would still require documentation and training, but not to the same degree.

10.0 images would not have this problem because they would be based on trixie. Context: https://wiki.debian.org/ReleaseGoals/64bit-time.

In short, we have to sacrifice compatibility for Debian or Ubuntu for Arm32 for .NET 9 due to this breaking change. Given the way our tags are setup, we think it is more important to preserve compatibility with Debian for .NET 9 and to align timing with Debian. Either way, .NET 9 is the release in question. This won't be fixed for .NET 8 and it will definitely be fixed by .NET 10.

Do folks think this is a better plan?

[richlander]

That matrix looks great to me. It's what I intuitively thought the matrix should be (since the correct glibc is in those OS versions) but didn't have the depth to validate that.

[richlander]

I'm going to continue with the breaking change notices, per this (the one proposed initially) plan. We'll get a build (as you suggest), test it, and go from there.

@sbomer

[agocke]

It would be good to confirm what the oldest glibc version actually is that has time64 armhf. Is it 2.34?

[jkotas]

Is it 2.34?

Yes, according to the docs https://www.gnu.org/software/libc/manual/2.34/html_mono/libc.html (look for _TIME_BITS ). Also, I have done some ad hoc tests earlier to confirm it.

[richlander]

Just to keep everyone on the same page, we're going to go forward with the initial plan. This side conversation was useful, but we decided to not adopt this counter-proposal.