Sometimes many packages need to change in some way at once. This may be to implement a new packaging policy, to use of new RPM macros, to switch URLs, or many other reasons. Here are some things to keep in mind before you start making mass changes to Mageia packages.
Contents
Get a consensus that it's a good idea
If you're about to change dozens (or hundred or thousands) of packages, you'd better be sure everyone is on board with the changes. Post your idea to the dev list first to get some feedback. It may be that other developers have different ideas about how to solve the issue or perform the change. It would be annoying if you had to revert your changes because you didn't think everything through. Only if the change is simple, noncontroversial and fairly contained should you skip this step.
Use a useful log message
It's more important that the change log make sense when it will be seen in hundreds of packages (double-check your spelling!). Consider if the change is worthwhile being seen in the RPM %changelog and use SILENT to hide the message if not.
Be careful!
You are about to change many packages using a single command, as easily as you would change a single package. You have the ability to break a large chunk of Mageia if you goof up. Try to avoid that by following these steps.
Check and recheck
After you make your mass change on your local system, check the diffs that result and look for problems. If you're not changing thousands of packages, you can easily scan over the diffs and look for patterns that don't look right. Did you use a regex in your change that failed to consider a case? Is your regex changing things that it shouldn't? Did you forgot to exclude/include commented lines from your change? Are you matching only lines in the correct section of the spec file? Is there a typo in your change? Use your brain's intuition as you scan for problems.
Test a subset
Choose a couple of packages at random and test them thoroughly. Build them locally, then submit them and make sure they still work in the build system. That verifies that the changes you made were sound.
Rebuilding
Does your change require that the packages be rebuilt? That's the only way your changes will make it onto end-user systems, but that's not needed for all changes. If you're only changing comments, then there's really no need to rebuild the package. If the change you're making doesn't result in any change to the RPM (e.g. switching from a deprecated build macro to a new one), then a rebuild isn't needed. If the change is only minor (e.g. fixing a typo), then you really don't need to rebuild it right away. Just wait for the next mass rebuild or version bump and your changes will be incorporated at that time, only be sure that you haven't just introduced a latent problem that will cause the next rebuild to fail and ruin another developer's day!
Bumping the release
If the packages need to be rebuilt right away, then the %mkrel needs to be updated before the package is submitted. It's best to do this at the same time as your other change to vaoid two submits for one change. There are scripts available to help do this.
Submitting packages for building
Submitting a large number of packages to the build system should be coordinated with the Sysadmin Team to avoid blocking the build system for other packagers. Packages should also be submitted in dependency order so packages are always built with their latest dependencies.
When dealing with perl packages, some of the same instructions as when Rebuilding packages for new perl may be applicable. Note that one can use the "magpie dwim" command to update all perl packages from CPAN.
When dealing with python packages, some of the same instructions as when Rebuilding packages for new python may be applicable.
Designing mass edits
Most mass changes will involve editing .spec files. The solutions in the Tools section below will bring many .spec files onto your local machine for editing. As you edit the files, keep the following in mind.
Macros
RPM macros can be used almost anywhere in the file. That means, for example, if you want to change the URL: line, the contents of the line might not be a straight text string but might include something like %{name} that is replaced as the file is used. You can use the rpmspec program to expand macros in .spec files for you so you can match lines, but you can't change them at the same time.
File sections
Sometimes changes should be limited to a specific RPM file section (e.g. only the %description section). The different RPM file sections are introduced with lines like %build and %install, but since macros can start with % as well, it's not trivial to look at only one section. A sed command like:
sed -i -e '/^%description/,/^%/s/Who/Whom/g' */*spec
that attempts to limit changes to the %description section(s) only could miss words if the description contains a macro (like %name) at the start of the line. Keep in mind that some sections can appear multiple times (for subpackages) and one (the main package information in a .spec file that also defines subpackages) has no section header.
Think about the implications of your mass change edit and try to err on the side of not breaking things.
Regular expressions
Regexes are a powerful means of matching text, but it's easy to create subtly wrong regexes. Here is an example sed command for changing the %make macro to %make_build that almost works but has some serious problems that are mentioned below:
sed -i -E 's@%make@%make_build@' */*spec
When designing them, keep the following points in mind to avoid a catastrophe that accidentally breaks thousands of projects:
- Consider if strings should be replaced no matter in which section they appear in the spec file. If not, try to limit them to the correct section. If only strings in the %build section should be touched, the sample command should be more like:
/^%build\>/,/^%(description|package|prep|build|install|check|files|clean|changelog|pre|preun|pretrans|post|postun|posttrans|transfiletriggerin|transfiletriggerpostun)\>/s@^%make\>@%make_build@
- Consider whether strings in comments should be ignored. sed actually allows a second regex to more generally match lines, making a total of three regexes in this example workflow that all have to work in harmony. Changing the example sed regex to:
/^#/!s@%make@%make_build@'
would cause it to ignore comment lines.
- Consider whether the position in the line affects whether or not it should be replaced. If only lines beginning or ending with the string should be changed, anchor it with ^ or $.
- Consider if the string might appear more than once in a line and if only the first or all instances should be replaced. By default, only the first is replaced; Use the 'g' suffix to sed's s command to replace them all:
s@\<tyop\>@typo@g
- Consider if the string should be matched case sensitively or not. GNU sed's 'i' suffix will match case insensitively.
- Consider if optional whitespace could affect the match. For example, a substitution like this should be case insensitive:
s@^Url:[[:space:]]*http:@Url: https:@i
- Try to keep the same whitespace as the original file. Don't replace tabs with spaces or vice-versa if you don't have to. The previous example suffers from that problem and will change any existing tabs to spaces. The following substitution fixes that:
s@^(Url:[[:space:]]*)http:@\1https:@i
- Consider the situation of partial string matches. The example below would would replace "%make" with "%make_build" but would also replace "%make_build" with "%make_build_build". This is also dangerous (and likely) when replacing URLs. Use "\<" and "\>" to match the beginning and end of words, or perform several replacement runs, replacing longer string instances each time.
Tools
Many text processing tools like perl, Python, awk and sed can be used to edit .spec files en masse, especially when combined with find and xargs, but following are some tools specialized for dealing with Mageia .spec files.
Youri::Package::RPM::Updater
This perl module (found in the perl-Youri-Package-RPM-Updater package) handles updating versions in spec files. Its most helpful feature when making mass changes is to bump the release number. Do that by running:
perl -mYouri::Package::RPM::Updater -e 'Youri::Package::RPM::Updater->new->update_from_spec("YOUR-PACKAGE.spec");'
That's what the rebuild-packages script uses to update the release number before a distro mass rebuild.
spec-tree
spec-tree is a set of scripts for maintaining a checkout tree of all Mageia spec files to make mass searching, updating and submitting doable. It is available in the spec-tree package starting with mga9.
.spec snapshots
Two daily snapshots of all .spec files in Cauldron are available as tarballs. https://pkgsubmit.mageia.org/specs/cauldron-no-svn-snapshot.tar.xz contains just the .spec files, while https://pkgsubmit.mageia.org/specs/cauldron-sparse-svn-snapshot.tar.xz also contains SVN control files so changes can be submitted (after an svn switch).
