Withdrawal of CentOS 8 test containers #4
Replies: 7 comments 38 replies
-
|
In some cases, you can work around the removal like this: ansible-collections/community.general#4132 |
Beta Was this translation helpful? Give feedback.
-
|
Why not, instead of complete withdrawal, change to CentOS Stream? New docker images are available at quay.io/centos/centos:stream8 or Indeed... it's a bad thing that in dockerhub there is no move for a long time. |
Beta Was this translation helpful? Give feedback.
-
You're describing regressions in the operating system, which I don't think should be the primary concern. Regressions happen in every distro, including RHEL. Depending on severity, when a regression happens in a RHEL minor release it may have to wait six months for the next minor release to be fixed. I've even seen some regressions take two minor verions (12 months) to get fixed. When it happens in CentOS Stream, the regression can be fixed in a matter of days or weeks. What Ansible should actually be concerned with testing against is the more common scenario of intentional changes in CentOS Stream that are planned for the next RHEL minor release, which are not regressions and will not be reverted. The most recent example of this was qt5 being rebased from 5.12 to 5.15. I understand qt5 isn't relevant to Ansible but it's just an example of the type of change that can happen in a RHEL minor version that is delivered in CentOS Stream first. When changes like this happen, it's valuable for Ansible to test against these changes by testing on CentOS Stream.
Yes, not testing against CentOS Stream is a loss in test coverage. Testing only on Fedora and RHEL (or RHEL clones) leaves a blind spot that could result in being caught off guard by intentional RHEL minor version changes. I'll also note that CentOS Stream 9 is already available, so that should be added to the testing matrix as well to start testing against the EL9 platform.
The purpose of CentOS has always been to provide an enterprise distribution. Rebuilding RHEL sources is how we accomplished that historically, typically with just 1-3 active maintainers at a time. Now we accomplish the same goal by having the entire RHEL organization build an enterprise distribution in the open, in partnership with the community. Yes, the development model is quite different, but the resulting distro is not very different at all. CentOS Stream is still is bound by the RHEL compatibility rules. It can only be as different as RHEL can be between minor versions. Currently only 5.38% of packages in CentOS Stream 8 have a higher version than the corresponding package in RHEL 8, and most of those are highly compatible increments such as 1.2.2 to 1.2.3. An additional 5.37% of packages have the same version but a higher release, which usually indicates additional backported patches. The rest of the packages are the same. This is why it is a continuation. The resulting distro is nearly the same, but the development model is more sustainable and actually allows for contributions to the operating system. |
Beta Was this translation helpful? Give feedback.
-
I'm not going to go into depth here currently, but you are making some assumptions about how the ansible core CI operates that are not correct. In practice, we simply wouldn't get the test coverage you imagine we would. |
Beta Was this translation helpful? Give feedback.
-
|
@sivel Is there anywhere you are willing to go into depth describing what I'm missing? Your other post in this discussion is also lacking in these details. I'd like to understand what assumptions you are referring to. |
Beta Was this translation helpful? Give feedback.
-
|
Before I answer that, which will still be relatively brief, I won't go down a rabbit hole to discuss whether or not our testing methodology should be changed. We just don't have the resources to go down that path. For all intents and purposes our testing environments are frozen, and updated on an infrequent basis, and are built with nearly all deps needed to run our tests. Should we test against Stream, it would be a frozen container, that may stay in that state for upwards of a year. And within the tests, we do not upgrade any fundamental functionality of the OS or testing environment. So python, dnf, selinux, kernel, etc... would never get updated. Effectively everything we test against is frozen and wouldn't actually benefit from a rolling release. So effectively for any given point of time, we would be testing 2 targets that are basically identical, and unchanging. We've largely designed our testing to eliminate anything rolling that could introduce instability in the testing environments. Note that we do test RHEL betas, and until recently we had RHEL9 beta in our CI matrix, until the introduction of a pulp bug that prevents our bootstrapping from installing gcc due to dependency issues and package sizes, required us to temporarily disable it. At some point, there is a necessity that others need to do their integrations with us, instead of core testing every conceivable platform and distribution. |
Beta Was this translation helpful? Give feedback.
-
|
CentOS Stream is not a rolling release. It has independent major versions (8 and 9 currently) and EOL dates. It would work just fine with what you've described here. As you've said the containers are frozen for long periods of time, so there is no more risk of introducing instability than any other operating system you're testing with. In fact there is less risk with CentOS Stream than most distributions because it is still bound by RHEL compatibility rules. If your argument is that CentOS Stream 8 is so similar to RHEL8 that it's not worth testing both of them, I buy that, but still would lean towards testing with CentOS Stream 8 instead of RHEL8. This way the testing containers would give you an accurate representation of testing against the EL8 platform for the longest duration in the future. RHEL8 is on a six month minor version cadence. That gives you a maximum of six months (if you generate the containers on release day) where it gives an accurate represents the EL8 platform. CentOS Stream 8 leads RHEL8 by about six months, so a CentOS Stream 8 testing container would give you closer to 12 months of accurate representation of the EL8 platform. To me that sounds like a better fit for your container regeneration schedule than RHEL8. No one is asking you to test every conceivable platform and distribution. CentOS 8 was previously included, so it's not a large ask to add CentOS Stream 8. Or more specifically to #3, recommend collection maintainers switch to testing against CentOS Stream 8, rather than dropping CentOS 8 outright. |
Beta Was this translation helpful? Give feedback.
-
|
I wanted to circle back here, to post an update from the Core team. We've had a lot of conversations about this over the past week. In the end, the Core team has decided to continue not testing CentOS Stream in our CI matrix. By extension of this declaration, the Core team will also not maintain a CentOS Stream image at this time. This decision has nothing to do with CentOS specifically, and purely with evaluating whether there would be Ansible Core testing benefit from adding these additional test targets, taking into consideration our CI and testing methodologies. Please don't try making this into something it isn't. CentOS is an important operating system, with an amazing history, and a bright future. I was personally involved many years ago as a member of the CentOS team, as its mirror admin. We continue to fully support CentOS as a controller, and as a target, and stand by our ability to support users of CentOS. I have made recommendations of creating a community repository, where community maintained images could be kept, with the potential of an associated Docker Hub or Quay organization for delivering those images. |
Beta Was this translation helpful? Give feedback.
-
|
@dmsimard - How can I help? |
Beta Was this translation helpful? Give feedback.
-
For you and others who may be interested in collaborating on community maintained images, please reach out to me on IRC (dmsimard) or Matrix (dmsimard:matrix.org). I can't yet point to a specific thing you could help with but I intend to set up a repository around which we can organize and work together. Thanks for your interest ! |
Beta Was this translation helpful? Give feedback.
-
That would really be awesome.
I'd like to see both Debian and ArchLinux available in CI. I don't expect the core team to maintain images for them for me though :) |
Beta Was this translation helpful? Give feedback.
-
|
@eRadical i'm making no assumptions no 'how things are viewed' as they don't factor into any of my commentary. I'm looking at it from a purely functional perspective, probably why I cannot understand the anger or frustration (or just being a Gentoo user it becomes the background). The decision to withdraw was also purely functional and has nothing to do with distro preference, views/reputations of distros or anything else. I would expect the opposite accusation to have at least the appearance of merit, that we kept CentOS this long because we work for RH. That said, I find what @dmsimard is doing great, I just don't think it should be imposed on others that don't see the need. |
Beta Was this translation helpful? Give feedback.
-
|
@sivel It looks like you've added "thumbs up" and "heart" and other reactions / emojis to your own comment. While I understand and agree with what you've said, I don't think you should "comment" your own comments like this. Please stop it. Actually, I wonder why GitHub allows people to do this. But anyway, you shouldn't imho. |
Beta Was this translation helpful? Give feedback.
-
|
The decision to not introduce testing for CentOS Stream 8 and CentOS Stream 9 into Ansible core and collection CI is a very disappointing one. Not just from the perspective of someone who would expect Ansible to continuously integrate against their primary platforms, but because I've encountered a breakage that seemingly only occurs in CentOS Stream 9 with a lookup plugin. I'm still trying to pin down what's going on, but this discussion worries me because there is evidence that there is already a gap in test coverage that is caused by not testing against CentOS Stream. |
Beta Was this translation helpful? Give feedback.
-
|
In the years we have been doing OS specific images for testing I can count in one hand the number of OS specific issues that it has prevented from being merged. Most issues we find that are related to systems (package manager, service manager, python versions) are present in multiple distros and OSs. If you have an issue with a lookup, please open a ticket with a reproducer and we'll look into it, I very much doubt your issue would have been apparent if we had CentOS stream in our test matrix. Not that we would be using 9 yet, since we just retired 8, so even if it was something our tests would have picked up, which I doubt, we would still be in the middle of setting up the line of images for testing. |
Beta Was this translation helpful? Give feedback.
-
|
Following up on the proposal for community-maintained testing images, please see this repository if you are interested: https://github.com/ansible-community/images (posting as a top-level comment as GitHub discussions thread would hide this comment unless manually expanded...) It is still heavily a work in progress but we are learning, experimenting and iterating on a framework to automatically build, test and publish images and have had success with three ansible-test images (debian bullseye, centos-stream8 and archlinux) as well as an execution environment that includes ansible 5. |
Beta Was this translation helpful? Give feedback.
-
|
Hey folks Firstly I think it's great that people are taking the time to think about the impact and share their thoughts. That's exactly what we wanted to achieve with news-for-maintainers/discussions and community-topics/issues. I thought it might be useful to update this discussion group on some of the achievements that Collection Maintainer has made, which means it's less important about the underlying OS we tests against. There was a major CI advantage when we moved to having dedicated GitHub repositories for Ansible Collections, and that's targetting CI. Previously when all 4,000+ modules were in the ansible/ansible repo it was we the tests for (say) MySQL just tested against whatever version of MySQL was included in the Operating System that ansible-test was run against. At this point in the past, it was very important that we had breadth (of OS) and depth (of OS versions) to test against. This is no longer the case. In the world of collection, we can do much more targeted testing, and in most cases, the host operating system no longer matters. Example 1: Application under test via container (GitHub Action)For example, in community.grafana we can test easily against three versions of Grafana using this GitHub Action integration:
strategy:
matrix:
grafana_version: ["7.0.6", "7.1.3", "8.1.2"]
services:
grafana:
image: grafana/grafana:${{ matrix.grafana_version }}Example 2: Application under test via custom install scriptWe do similar with Zabbix, though that uses a custom script to install the needed version, rather than using a container image. Example 3: Complex - Multiple applications under test via GitHub ActionFor PRs against community.mysql we have a more complex CI setup, that allows us to tests against a matrix of Where we need breadth and depthThere are some cases where we do need breadth (of OS) and depth (of OS versions), that's generally where we use Azure Pipelines (which gives us access to non-docker OS and Windows. This is needed for Collection repositories where we aren't testing against a specific thing, community.crypto is a good example of this, as we need to tests against different Crypto libraries. |
Beta Was this translation helpful? Give feedback.
-
|
I'd be interested in where folks think we might have potential gaps in testing so we can work out how to address them. |
Beta Was this translation helpful? Give feedback.
-
Re:
I don't have any actual discussion around this, but we've set the expectation of using discussions for conversations around new issues, so I'm opening this one to see how it goes and to see if the issue will include a backlink to this automatically.
Beta Was this translation helpful? Give feedback.
All reactions