Fixes #35959 - Use structured APT for deb content

[quba42]

RFC: https://community.theforeman.org/t/rfc-transitioning-katello-to-structured-apt-deb-content/34262
Issue: https://projects.theforeman.org/issues/35959
Replaces: #10420
Requires: candlepin/subscription-manager#3223

Note that the subscription-manager packages for Debian and Ubuntu published on http://oss.atix.de/ are already patched to support these changes.

[quba42]

This "review" (and the below comments) aims to explain the design approach for this PR.

Regarding the current state of this PR:

• I have tested the new "structured APT" functionality as follows:
  • Created an APT repo, synced it, and added it to a CV version and LCENV. Registered a host and checked the APT repo components and distirbutions (structure information) make it all the way into the repo file on the registered host, as appropriate for the environment it is in. This works.
• For deb content, the current state does not yet update the corresponding KatelloContent, whenever some candlepin content is updated. I have not found this to impact the functionality, but I will continue to work on this for the sake of consistency and to avoid problems down the road.

In my local dev environment I currently see two test failures relative to master branch:

ktest test/actions/pulp3/orchestration/export_test.rb
ktest test/lib/tasks/pulpcore/repository_vcr_test.rb

The failing VCR test does not turn green from re-recording the cassette. It complains about a call to POST https://localhost:23443/candlepin/environments/1/content, but only records calls to Pulp.

The other test appears to attempt calling a pulp_deb API endpoint from a Yum API object, which does not make any sense. I was so far unable to figure out why this is, but I think it is likely an issue with the test rather than the code (not 100% sure though).

What I am hoping for at this point:

It would be great to receive some initial feedback if the design approach for this feature is sound. In particular, an initial review should focus on potential risks to non-deb content types. The idea is that from the point of view of yum and other non-deb types, this is a pure refactor, that does not change any outcomes! Thanks in advance for any and all input!

[quba42]

The latest push adds the following fixes:

• Fixed the test/actions/pulp3/orchestration/export_test.rb test (by fixing the code).
• Consolidated the several settings to enable this feature into a single setting.
• Fixed two rubocop errrors.

Remaining ToDo's:

• Update the rake task to migrate existing content to structured mode.
• There is still a failing test in ktest test/lib/tasks/pulpcore/repository_vcr_test.rb.
• Manually test more deb content edge cases.
• Add new test coverage for the new deb features.

[ianballou]

There is still a failing test in ktest test/lib/tasks/pulpcore/repository_vcr_test.rb.

This is the reference used for rerecording Candlepin VCR tests:

Running the glue tests: record=true mode=all ktest ~/katello/test/scenarios/scenario_test.rb (save some time by doing other testing concurrently)

I noticed that test uses scenario_support.rb, which has its own record definition:

  def record(name, options = {})
    VCR.insert_cassette("scenarios/#{name}", options)
    User.current = user
    yield
  rescue
    raise
  ensure
    VCR.eject_cassette
  end

This is kinda tricky because that test may need to do both candlepin and Pulp recordings at the same time.

Another idea would be to stub out the live Candlepin bits.

[quba42]

I will be on PTO from 2024-07-16 to 2024-07-28. During that time @m-bucher will be my replacement for questions regarding this PR. Before I go, I want to leave some pointers on how one can test/review this feature:

For non-deb content types (or deb with structured APT disabled)

All workflows should produce the same outcomes as before. Since some of the codepaths to get to those outcomes have changed slightly, carefull review and broad testing is necessary.

How to test the new structured APT features

• First, you will need to enable the feature under "Administer > Settings > Content > Enable structured APT for deb content". This must be done before adding any deb type content, since the rake task for migrating existing content is not ready yet.
• Next, add some Debian repositories and populate them with content. Some important cases are:
  • The empty deb repo (containing no packages)
  • The upload deb repo (no Upstream URL, upload a .deb package to it)
  • Synced content (preferrably with multiple distributions and components set) Tip: Use on_demand for speed.
• Add these repos to some CVs, CCVs, LCENVs!
• Create an activation key, and register a Debian or Ubuntu test host as described here: http://oss.atix.de/
• Observe the resultant /etc/apt/sources.list.d/rhsm.sources. Pay particular attention to the configured Suites and Components. Does this match what you expect to be in the CV version the host is consuming?
• On a sync repo, change what Distributions/Components should be synced, re-sync, and create and promote a new CV/CCV version. Does running subscription-manager repos cause the values in /etc/apt/sources.list.d/rhsm.sources to be updated on your registered hosts?
• Smart proxy sync. The published repos on the proxy should have the same structure as on the server, and registered hosts should use the same Suits and Components in /etc/apt/sources.list.d/rhsm.sources compared to the same environment on the server.

[ianballou]

Code structure is generally looking fine to me, just one small comment. I need to give it a test now.

[ianballou]

Should this PR still be in draft form?

[quba42]

Should this PR still be in draft form?

I just came back from PTO/am continuing on this PR now, currently there are three things I am waiting on to undraft:

• The one failing test I have not yet figured out
• The enable/disable rake task I have yet to update
• It looks like there is a bug in the implementation where activation keys are no longer showing any "repository sets" for deb content.

[quba42]

Update: I rebased to latest master and re-recorded VCR cassettes (because of conflicts).

For me all tests are now locally green.

I have two remaining TODOs before un-drafting:

• The "repository sets" view on activation keys for deb content is broken because of a broken query.
• The enable/disable rake task needs to be reworked.

[quba42]

Update: "repository sets" query for deb content (with structured APT enabled) is partially fixed, but needs to be refined further.

[quba42]

Update: To my best knowledge the feature itself is now ready for final/full review.

I am currently reworking the enable/disable rake task.

[quba42]

Update: Enable/disable rake task is 70% done. There are three TODOs two of which should be pretty trivial. Since I won't be working tomorrow, and Monday is full of meetings, I estimate I will be ready to undraft on Tuesday.

[ianballou]

I haven't been finding time lately to test this -- @quba42 if someone is able provide a testing report with steps taken from your team, it'd make it easier for us to get more eyes on it.

[quba42]

Update: Unfortunately I found that handling of the Library environment is currently broken. (Non-Library environments work fine). I am working on fixing this. I will also add a testing report what has been tried already.

[quba42]

Update: I have fixed the handling of Library content view environments. I have also fixed the enable/disable rake task to the point where it appears to be working in both directions, at least for my relatively simple initial test case (more testing is needed).

This means I now have a state that is to the best of my knowledge both complete and working. I will now rebase to latest master branch and re-record vcr_cassettes. Then I will undraft the PR.

[quba42]

Update: Pushed a minor cosmetic change. 😉

[jeremylenz]

Some minor nitpicks below.

@parthaa Can you take a look specifically at the changes to Dynflow actions and see if they look good to you?

[quba42]

Update: The latest state incorporates the latest set of suggestions by @jeremylenz

[quba42]

Incomplete list of things I have tested (split up into several parts):

Testing the change with the feature disabled

The point of these tests is to show that this change is a pure refactor for content types other than deb (as well as for deb with the feature disabled).

For some time now, all our internal builds have been using this change with the feature disabled. This includes large production like systems that were upgraded to this state. This means everything that was done on these systems has tested the "disabled" case. Some things that are explicitly worth mentioning:

• Sync, Publish, and Promote for various content types including deb and yum for both CVs and CCVs.
• Export/Import of yum and deb content.
• Backup and restore of yum and deb content.
• Host deployments and host registration, installing content on hosts.

For all of these reasons, I am very confident the refactor part is just that.

Testing the enable/disable rake task

• Tested the rake task in both directions in forklift using bundle exec rake katello:migrate_structure_content_for_deb (the direction is determined by the UI setting).
• Tested creating some deb content (dsiabled), then flipping the setting to enabled, creating more deb content (enabled), then migrating just the disabled content using the rake task.
• Tested re-running the rake task with nothing left to migrate in both directions (prints a message and does nothing further).
• Tested the rake task on a large production like installation with a large amount of deb content. (More than 40 root repos of various sizes published into various CVs, CCVs, and promoted to various environments).

Testing of the feature itself (deb content with the feature enabled)

• Tested Sync, Publish, and Promote for deb content with multiple components and releases/distributions.
• Tested re-sync after dropping a component or release/distribution, and promoting that.
• Tested registering a host using an activation key.
• Tested overriding repos on an activation key to disabled/enabled.
• Tested changing the content source for a content host.

Detailed description of a front to back test case.

• Start a forklift without any deb content on it.
• Enable the setting in the UI.
• Create product P Ubuntu 22.04 jammy
  • Create repository "R Ubuntu 22.04 jammy client"
    • Upstream URL: http://oss.atix.de/Ubuntu22LTS/
    • Releases/Distributions: stable
    • Components: main
    • Architectures: amd64
  • Sync
• Create product P_structured_apt_tests
  • Create repository R_deb_empty
  • Create repository R_deb_upload
    • Get example package: wget ftp.de.debian.org/debian/pool/main/v/vim/vim_9.0.1378-2_amd64.deb
    • Upload it
  • Create repository R_deb_multi_dist
    • Upstream URL: https://fixtures.pulpproject.org/debian/
    • Releases/Distributions: ginnungagap ragnarok
    • Components: asgard jotunheimr
    • Sync it.
• Create LCENV DEV
• Create CV CV_structured_apt_tests
  • Add all four test repos
  • Publish and promote new version
• Create activation key AK Ubuntu 22.04 jammy DEV
• Register an Ubuntu host using the key and the instructions from here: http://oss.atix.de/html/ubuntu.html
• Check the /etc/apt/sources.list.d/rhsm.sources
  • Check the Suites and Components fields. They should contain things like ginnungagap ragnarok and not just default or all.
• Now go back to the repo page for R_deb_multi_dist
  • remove ginnungagap and asgard
  • re-sync
• Verify that nothing changes on your registered host from the sync alone (after subscription-manager repos)
• Publish and Prmote a new CV version
• Verify that your host no longer has ginnungagap and asgard configured (after subscription-manager repos)

[m-bucher]

Had a meeting with @quba42 just now and looked through/discussed the latest changes again.
As far as I am concerned, this is good to go now 👍

[sjha4]

I did some testing for non-deb content and I was able to run the migrations and able to sync different non-deb repo types, have their content be setup correctly in candlepin and artifacts created in pulp..Pulp DB looked rightly setup and registration workflows went through fine and a subscribed host is able to consume the distributed content. Based on the code, most of the tasks updated look to be running successfully.

Note that this is not an exhaustive test but based on the code changes, ATIX testing, PRT testing and other reviews, this looks in a good state for merge.

ACK 🎉

[quba42]

Thanks a ton to everyone involved!