RPM: Group & Module Information missing

[Starbix]

I'm currently evaluating repomanager as a replacement for Spacewalk since it only causes trouble for us.
We're almost exclusively using Oracle Linux 8 and rely on DNF module streams. Repomanager doesn't download information about modules and thus if we were to replace Spacewalk with repomanager our machines would break.
Example: We're using the default 5.26 perl module stream and thus we only get perl 5.26 packages. Repomanager doesn't have module information so our hosts would try to install perl 5.32, breaking lots of packages.

This module information is stored in *-modules.yaml.gz files (see https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/repodata/). Other files may also be necessary for all functionality.

I've tried simply downloading all the files present in repodata/ to the corresponding directory in /home/repo and the client was able to get information about modules.

I'm not quite sure of the architecture of repomanager, but having the modules file present on disk and in the repomd.xml should suffice I think. It would be amazing if this could be implemented!

If modules aren't working the module list looks like this (note @modulefailsafe):

[root@update-test-harbor ansible]# dnf module list
Last metadata expiration check: 0:02:26 ago on Thu Jul  4 13:56:08 2024.
@modulefailsafe
Name                                                           Stream                                                        Profiles                                                                    Summary                                                                                      
httpd                                                          2.4 [e]                                                       common, devel, minimal                                                      Apache HTTP Server                                                                           
perl                                                           5.26 [e]                                                      common, minimal                                                             Practical Extraction and Report Language                                                     
python36                                                       3.6 [e]                                                       build, common                                                               Python programming language, version 3.6                                                     
python38                                                       3.8 [e]                                                       build, common                                                               Python programming language, version 3.8                                                     

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

If it does it looks like this:

[root@update-test-harbor ansible]# dnf module list
Last metadata expiration check: 0:02:26 ago on Thu Jul  4 13:35:32 2024.
oracle8-appstream repo on repomanager-test.example.com
Name                 Stream          Profiles                            Summary                                                                                                                                                                                                                      
389-ds               1.4                                                 389 Directory Server (base)                                                                                                                                                                                                  
ant                  1.10 [d]        common [d]                          Java build tool                                                                                                                                                                                                              
container-tools      ol8 [d]         common [d]                          Most recent (rolling) versions of podman, buildah, skopeo, runc, conmon, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and updated as frequently as every 12 we
                                                                         eks.
container-tools      1.0             common [d]                          Stable versions of podman 1.0, buildah 1.5, skopeo 0.1, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.                             
container-tools      2.0             common [d]                          Stable versions of podman 1.6, buildah 1.11, skopeo 0.1, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.                            
container-tools      3.0             common [d]                          Stable versions of podman 3.0, buildah 1.19, skopeo 1.2, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.                            
container-tools      4.0             common [d]                          Stable versions of podman 4.0, buildah 1.24, skopeo 1.6, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.                            
eclipse              ol8             java [d]                            An open, extensible IDE and application platform                                                                                                                                                                             
freeradius           3.0 [d]         server [d]                          High-performance and highly configurable free RADIUS server                                                                                                                                                                  
gimp                 2.8 [d]         common [d], devel                   gimp module                                                                                                                                                                                                                  
go-toolset           ol8 [d]         common [d]                          Go                                                                                                                                                                                                                           
httpd                2.4 [d][e]      common [d], devel, minimal          Apache HTTP Server                                                                                                                                                                                                           
idm                  DL1             adtrust, client, common [d], dns, s The Oracle Linux Identity Management system module                                                                                                                                                                           
                                     erver                               
idm                  client [d]      common [d]                          OL IdM long term support client module                                                                                                                                                                                       
inkscape             0.92.3 [d]      common [d]                          Unsupported. Obsoleted by the inkscape1 package                                                                                                                                                                              
javapackages-runtime 201801 [d]      common [d]                          Basic runtime utilities to support Java applications                                                                                                                                                                         
jaxb                 4 [d]           common [d]                          Jakarta XML Binding API and Implementation                                                                                                                                                                                   
jmc                  ol8 [d]         common [d], core                    Java Mission Control is a profiling and diagnostics tool for the Hotspot JVM                                                                                                                                                 
libselinux-python    2.8             common                              Python 2 bindings for libselinux

...... 

EDIT: the same goes for DNF groups

I found the corresponding documentation from Red Hat:

You should not use createrepo on a RHEL 8 repository or you will delete the module data. reposync was updated to download all of the repodata for you making a local repository, a single command.
How to create a local mirror of the latest update for Red Hat Enterprise Linux 8 and 9 without using Satellite server?

Raw

# reposync --download-metadata --repo <repo-id>

If you use createrepo on a RHEL 8 repository, you must also:
add the comps.xml with createrepo for groups
add the updateinfo.xml from cache for security package identification
add the modules.yaml from cache for modules, and you have to do this or you will encounter massive dependency issues.

Raw

# rm -rf /var/cache/dnf
# yum module list && yum updateinfo
# reposync --repo <repo-id> --download-path=/var/www/html

### With the repository downloaded, we now need to build the repodata. We should be sitting in the root of the repository.

# cd /var/www/html/<repo-id>

# cp $(find /var/cache/dnf/<repo-id>* -name *comps.xml) comps.xml
# createrepo -g comps.xml ./

### Find and add the module data  **NOTE** createrepo_c-0.16.2-2.el8 and later will automatically detect and include the modules.yaml, if present in the repo directory.

#  gunzip $(find /var/cache/dnf/<repo-id>* -name *modules.yaml.gz) -c > modules.yaml
# modifyrepo modules.yaml repodata/

###Find and add the security data
# gunzip $(find /var/cache/dnf/<repo-id>* -name *updateinfo.xml.gz) -c > updateinfo.xml
# modifyrepo updateinfo.xml repodata/

[lbr38]

Well okay that seems to be another complex feature request, I never heard of modules and groups to be honest and I can't just remove createrepo.

Like I said in some other issues I don't have time these days to work on such hard features which require a lot of changes and tests. I'm coding Repomanager on my own personal time and such feature could require a complete day to be implemented and tested. I'm sorry but don't expect it to be released any day soon.

I don't know about Spacewalk but if you really need a solution that can fully integrate into your IT system, maybe have a look at paid solutions (i.e. Redhat Satellite & co)?

[Starbix]

Yeah I totally get that. Depending on the decision of Spacewalk's replacement I may be able to implement this change on company time, if you are interested in PRs. We are looking for something that is less complex than Spacewalk (its performance is also really bad and thus hinders our workflow) and without a subscription.

Anyways, enjoy your summer:)

[lbr38]

I am open to PRs, feel free to contribute if you wish, others have already done so :)

[lbr38]

It's been a rainy day today and I took some time to work on this. I think I should be able to provide a "beta" release of this feature for the next week but I will need your help to do some tests because I am very not comfortable with those modules and groups things.

I will keep you in touch.

[lbr38]

I think I have finished developing the feature.

On your side, could you set up a new docker instance for tests, based on the devel branch? You will have access to the 4.4.0 release version and the feature. It would help me a lot if you could confirm that dnf groups and modules are working as expected before merging and realeasing.

To build a container from the devel sources, you have to specify env: devel in the build arguments. E.g. with docker-compose:

---
version: "3"
services:
  repomanager:
    container_name: repomanager
    image: repomanager
    build:
      context: .
      dockerfile: Dockerfile
      args:
        env: devel # <--- here
        fqdn: repomanager.example.com # change to your FQDN
        max_upload_size: 32M
    restart: always
    ports:
      - "8080:8080"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/lib/docker/volumes/repomanager-data:/var/lib/repomanager
      - /var/lib/docker/volumes/repomanager-repo:/home/repo

Thanks for the help.

[Starbix]

Wow thanks so much!
Modules, groups and updateinfo are all working now!!
Setting "Task execution memory limit" to 1GiB also seems to fix the issues with XML parsing for Oracle Linux repos.

[lbr38]

Thanks for the feedback, I will merge this.

Note: I just opened a discord channel here https://discord.gg/34yeNsMmkQ if you need quick help or ask questions (french or english spoken). In some case I think it can be easier to chat for quick debugging.