From Mageia wiki
Revision as of 22:11, 16 September 2017 by Ngompa (talk | contribs) (Fix last updated datestamp)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Summary

Package DNF and associated tools to enable usage of DNF as an alternate repository manager in Mageia. Start generating the necessary repository data for DNF to be fully functional on Mageia.

Owner

  • Name: Neal Gompa
  • Email: ngompa13@gmail.com

Resources

This will require efforts by packagers to get DNF, its plugins (core and extras), and distribution-side tooling into the distribution. Additionally, the buildsystem's repository generation step will need to generate metadata for DNF to consume if we want to enable usage of DNF as an alternate repository manager. A Metalink system (could be deployed using MirrorBrain or MirrorManager2) will need to be implemented for proper mirror registration and usage.

I intend to be heavily involved in efforts to get DNF into Mageia and into a working state.

Current status

  • Targeted release: Mageia 6
  • Last updated: 2017/09/16
  • Percentage of completion: 90%

Detailed Description

As of Fedora 22, Yum has been replaced with DNF as the default repository manager. Consequently, Yum is deprecated and in maintenance mode now, with all efforts being focused on DNF. DNF is powered by libsolv through hawkey for dependency resolution and uses libcomps and librepo for processing metadata. Unlike its predecessor, DNF has a well-defined API that can be used to integrate with other applications and easily extended through its APIs.

Why it would be good for Mageia to include it

Mageia currently includes Yum, though it's not really functional or useful. The Fedora package tools (like mock) now require DNF in current releases. In order to keep those useful when those packages are synced/updated, we should have DNF and its associated plugins and tools available and working.

The availability of support for DNF compatible metadata will also make it easier for people to build packages targeting Mageia with tools like mock on other distributions, where the tools are available and minimal configuration is required. Critically, DNF is actively developed and supports Python 3 (unlike Yum, which is only minimally supported and not able to work in a Python 3 only world). DNF is actively maintained as part of the rpm.org project, meaning that it receives fixes and features as soon as RPM itself has it. With DNF, Mageia would have a dependency resolver sharing common technology with Fedora and openSUSE, opening more opportunities to collaborate with them and benefit from shared efforts.

It will also enable usage of software on top of DNF with Mageia (such as rpm-ostree's toolbox software, recent versions of Spacewalk and Korora's Canvas) and development of Mageia software on top of DNF.

Why not replace urpmi with DNF now?

With the successful implementation of this feature, it would be technically possible for users to use DNF exclusively and never use urpmi. And because DNF is maintained as part of the upstream rpm.org project, it would greatly reduce the burden of the few people we have maintaining the tooling for Mageia if perl-URPM and urpmi were dropped. It would allow them to work on other matters for Mageia.

However, there are a few reasons for not proposing fully replacing urpmi right now:

  • The installer still requires perl-URPM and thus urpmi. If the installer code could be reworked to use DNF (through the CLI, dnfdaemon, PackageKit, or whatever), then this issue falls away.
    • urpmi could still be on the install image without being on the installed system, as a stopgap, if desired. This is even easier if the installer uses perl-URPM without calling the urpmi CLI at all.
  • rpmdrake, like the installer, requires perl-URPM and urpmi. There are a couple of ways to deal with this problem if removing urpmi is desirable:
    • rpmdragora (the successor to rpmdrake) could be reworked to use DNF (through dnfdaemon, PackageKit, or whatever)
    • rpmdrake/rpmdragora could be dropped in favor of relying on PackageKit based front-end UIs primarily and have Yumex-DNF for the non-PK based GUI frontend
  • The build system requires urpmi. The build system currently uses iurt for building packages, and iurt uses perl-URPM and directly utilizes urpm* commands. Note that this is not really a blocker to having DNF be the exclusive mechanism on releases, as this is infrastructure-side only. There are a few ways to deal with this problem if removing urpmi is desirable:
    • Rework iurt to use DNF
    • Rework the build system to use mock, which will be fully functional as part of the implementation of this feature
    • Switch to a build system that doesn't depend on iurt, such as Koji (which uses mock)
  • It may be desirable to have the urpm* command set available for admins and scripts to smoothly transition over to DNF. The dnf-URPM project is designed to address this, but needs work to support all existing urpm* commands.

Generally speaking, the only major blocker to shipping DNF as the only package manager involves work to develop dnfdragora to replace rpmdrake (or dropping the custom GUI frontend for GUIs that have PackageKit based frontends central to their environments). Depending on what you consider minor/major, getting the installer to use DNF instead of perl-URPM/urpmi may either be a minor issue or a major blocker.

Judgements of the difficulty of accomplishing this would involve discussions with the developers/maintainers of these respective systems. That said, the current scope of this feature is easily achievable. If it were decided to expand the scope to replace urpmi in its entirety, obviously much more work would be involved.

Test case

To test this, metadata for DNF will need to be generated for repositories that DNF can consume. The following actions should be tested to work:

  • Installing packages
  • Updating packages
  • Removing packages
  • Getting information on packages
  • Searching for packages
  • Searching package/repo data for files
  • Get a list of transactions
  • Get information on a transaction
  • Undoing a transaction
  • Redoing a transaction
  • Building packages using mock+DNF

Tasks

  • Package up all the necessary software
  • Alteration of the build system to automatically generate the required repository metadata
    • Optionally (though preferred), generate comps.xml data to translate task-* packages into installable/trackable groups
  • Deployment and configuration of a mirror management service that generates metalinks/mirror lists for DNF to use to select mirrors

Software / Packages Dependencies

DNF packages

Repository metadata generation packages

Packages in this section have to be available in Mageia 5 (infra_5) and Cauldron, in order for the automatic metadata generation to be enabled

Packages for hif PackageKit backend

  • libhif - reuses dependencies and data from DNF (entered Cauldron on 2015-10-04)
  • PackageKit >= 1.0.8 - Cauldron is currently at 1.0.6 and needs to be upgraded (at time of writing, 1.0.10 is available) (updated in Cauldron on 2015-10-10)
  • PackageKit must be rebuilt to enable the hif backend (done in Cauldron as of 2015-10-10)

PackageKit is now using the Hif backend exclusively in Cauldron, as of 2016-03-28.

Since 2016-11-26, PackageKit transitioned from the Hif backend to the DNF backend, and libhif was replaced with libdnf.

Packages for the DNF version of Yum Extender (a non-PK GUI frontend)

  • dnf-daemon - exposes DNF API through DBus (entered Cauldron on 2015-10-25)
  • yumex-dnf - uses dnf-daemon (entered Cauldron on 2015-10-25)

Packages need updating to use DNF

  • mock - Cauldron version is at 0.9.14, while current version is 1.2.13 at the time of writing (updated in Cauldron on 2015-10-10)
    • Mageia mock configs must be written and added to mock package once DNF and its core plugins are in and working (metadata is required for this)
      • Initial configs submitted to the Fedora Buildsys group ML for inclusion into mock. (included in mock devel branch 2016-03-14)
        • Updated configs submitted and patches backported to current mock in Cauldron as of 2016-05-30.
          • Once the next mock release arrives with Mageia configs, Fedora Copr will add the Mageia Cauldron (and later Mageia 6) target for building and generating repositories. Once this happens, the dnf copr plugin will be built for Mageia. More info about this is in this ML post.
  • yum - The yum-deprecated patch needs to be applied to the package. (fixed in Cauldron on 2015-10-25)

Packages for enabling (safer) distribution upgrades through DNF

  • dnf-plugin-system-upgrade (entered Cauldron on 2016-01-24)
    • Backport of this with dnf+dnf-plugins-core will be made available for mga5->mga6 upgrades
  • PackageKit >= 1.0.8 - Cauldron is currently at 1.0.6 and needs to be upgraded (at time of writing, 1.0.10 is available) (updated in Cauldron on 2015-10-10)

Packages for providing Mageia repository configuration through DNF

  • mageia-repos - New package providing the repository configuration and GPG keys for Mageia for DNF, which will be Required by mageia-release.
    • Package now exists, providing the GPG key but not yet repository files.
      • UPDATE: A temporary workaround until metalink setup is done has been implemented as of 2016-05-30. Thus, mageia-repos now provides working repository configuration.

What could disrupt development of this new feature

If the automated metadata generation can't be implemented (which involves backporting some tools to Mageia 5) or all the packages can't be built and functional in Cauldron, then this feature cannot be fully implemented.

Planning

Early inclusion into Mageia 6 would permit the widest array of testing to ensure the processes for automated metadata generation and usage of DNF as a repository manager are functional.

Contingency

Since this feature is not intended to displace urpmi as the repository manager, the failure or success of the completion of this feature will not hamper the development/release of Mageia 6. If it fails to be ready for Mageia 6, release what we have and continue on for Mageia 7.

Release Notes

The release notes should indicate that Mageia 6 includes support for DNF as an alternative repository manager to urpmi. It should also indicate that PackageKit is now using the "DNF" backend, which promises greater reliability for PackageKit based frontends like GNOME Software, Apper, etc.

Documentation

Packager comments

  • ovitters: Please just upgrade all package versions which need upgrading
    • ngompa: OK
  • ovitters: Instead of applying yum-deprecated patch, just have dnf obsolete yum (yum never really worked so no need to keep it)
    • ngompa: yum is needed for mock to function for targeting EL and FC21