This page describes tools to facilitate the management of Mageia ISOs for pre-release testing. It also details the use of rsync directly for this purpose. It is a supplement to the "Pre-release ISO testing" page.
- 1 Introduction
- 2 MageiaSync
- 3 Dorsync
- 4 Onesync
- 5 Rsync directly
Mageia ISOs are hosted on a server: bcd.mageia.org/isos.
Here there is a directory for each Release e.g.
mageia4-final mageia4-v2 mageia5-alpha1 mageia5-alpha2 mageia5-beta1
Each Release comprises all the ISOs, e.g. for mageia5-beta1:
Mageia-5-beta1-LiveCD-GNOME-en-i586-CD Mageia-5-beta1-LiveCD-KDE4-en-i586-CD Mageia-5-beta1-LiveDVD-GNOME-i586-DVD Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD Mageia-5-beta1-LiveDVD-KDE4-i586-DVD Mageia-5-beta1-LiveDVD-KDE4-x86_64-DVD Mageia-5-beta1-dual-DVD Mageia-5-beta1-i586-DVD Mageia-5-beta1-x86_64-DVD
In fact each so-called ISO is a directory which contains not only the actual ISO image file *.iso, but additional files like date, checksums. For example (the Classic DVD):-
DATE.txt Mageia-5-beta1-x86_64-DVD.idx Mageia-5-beta1-x86_64-DVD.iso Mageia-5-beta1-x86_64-DVD.iso.md5 Mageia-5-beta1-x86_64-DVD.iso.sha1
or (a Live ISO):-
DATE.txt Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.md5 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.sha1 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.langs Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst.full Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst.leaves Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst.names
Note well that both the ISO directory and subordinate file names incorporate the Release. This has consequences for you depending on which rsync tool you use.
The ISOs you are testing reside in a local directory of your choice. To keep them synchronised with those on the server, these ISO directory (mostly) and subordinate file names must be identical. Those on the server change at each new release (in effect, the download source changes). How you keep your local copies in line with those on the server is described for each of the tools - some do it for you, others you have to do it yourself.
Why have tools?
Both the initial download (priming) & subsequent updating of your local ISOs are done using rsync, which has the powerful & important ability to download only the changed parts of files, noteably the large ISO images, greatly reducing download times. The use of rsync is very exact: it requires great care to make it do just what you want, and the long ISO directory names make difficult parameters. If you get it wrong, you will probably download entire ISOs you do not want. Its specific usage is covered later in this page. The tools described beforehand will make your life much easier.
Global rsync information
Any use of rsync in our context includes certain basic parameters, which are required by all the methods described, and defined for them in different ways:
- The main ISO source, the server: bcd.mageia.org/isos. This is permanent.
- An rsync 'user': e.g. isoqa. This remains constant over the long term, but *can* be changed, in which case it is announced on the QA MailList.
- An rsync password: this is given out only by private e-mail, initially and when it periodically changes.
Applies to all the tools: despite its amazing capabilities, it seems that any line break stops rsync: it does not recover/resume after the break. You probably will have to abort it. Re-running the synchronisation, it will quickly catch up to where it got to, and carry on. Thus, if necessary you can interrupt an ongoing transfer without significant penalty when you re-do later.
mageiaSync is a tool to get one or more ISOs images of Mageia, from testing repositories or from mirrors. It is the main tool for ISO testers, the only one with a GUI. It manipulates ISOs under their full long (and evolving) names. It has a facility (after a new Release) to rename local ISO directories & files - on demand.
- Lists remote and local ISO directories and image files.
- Allows the selection of one or more images to download/update.
- Verifiy the signatures of MD5 and SHA512 sums provided and performs checks of them.
- Provides a tool for Release changes, i.e. renaming directories and files with the new name.
An RPM package mageiasync is available for Mageia 5 and later.
To launch it, there is a menu entry "MageiaSync" under System Tools. From a console, simply type:
- User: this is the rsync username e.g. isoqa .
- Password : the current rsync password.
- Source: The download ISOs source. Normally leave this empty for the main Mageia ISO source; otherwise give a mirror address with the path to the release.
- Release: The drop-down list shows all the releases currently available, e.g. mageia5-alpha2. Choose the one that interests you, normally the latest.
- User [as above]
- Password [as above]
- Source [as above]
- Destination: your local directory where the ISOs will be stored. Click it to raise a file picker dialogue [beware: this may be hidden beneath other windows].
- Bandwidth limit: keep 0 if you don't want to apply limit.
Menu File->Preferences enables you to edit these values subsequently.
[Config file is ~/.config/Mageia/mageiaSync.conf]
If the top 'Release' frame is void, then some information are wrong, use File->Preferences to adjust them, then File->Update list.
Clicking OK leads to the main dialogue below.
Regular use: the main window
- The Remote directory frame lists all the ISOs at the server for the current release. Select the one(s) you want. If you do not already have it, it will be downloaded; otherwise it will be updated. The Select All button does what it says - if you really want the whole set.
- the Local directory frame identifies your local ISO umbrella directory, and lists the ISOs you currently have in it.
- The Do sync button launches the synchronisation. This can take hours, but if resumed later recovers very efficiently; so do not be frightened of using the Stop button if you need to.
After the download completion, the signatures of MD5 and SHA512 sums stored locally are verified, and the sums are computed and checked versus the ones stored in files. If signature files are provided, the signature is verified and the sums are matching, 'OK' will be displayed. If 'Failed' is displayed, thus one of the previous conditions is not true. See this example when MD5 sum is signed, with the icon showing that the signature is OK.
When the Release changes
At a new release, all your local ISO directory and subordinate filenames must be updated to be the same as their new equivalents on the server. From the menus, Sync-> Rename archives
shows the Rename release dialogue.
- Base directory: shows your current ISO umbrella directory; click if you want to change it (unlikely: you would end up downloading entire ISOs).
- Old release: e.g. Mageia-5-alpha2
- New release: e.g. Mageia-5-beta1
OK and it is done! You will have the number of directories and files affected by the change.
This command line shell script was the main tool before the advent of Mageiasync. It originally assumed that you juggled *all* the ISOs, but now allows the exclusion of those you do not want. It manipulates ISOs under their full long (and evolving) names. Apart from the main business of synchronising your ISOs, it will make bootable ISO USB sticks of them if you want - the only rsync tool to do so.
- Operates implicitly on the entire ISO set, but ...
- Allows specific ISOs to be excluded.
- Checksums the *.iso image files.
- Can create bootable ISO USB sticks both after synchronisation, and independantly.
dorsync can be easily downloaded via
Put it where you want (ideally in a directory in you PATH) and make it executable with:
$ chmod a+x dorsync
Before you use dorsync, you must define, by editing the start of it, several values:
############################################ # Enter a few variables. # # location is where you stored the ISOs # # release is the directory on the server # # user is the rsync login username # # password is the rsync password # ############################################ location="" release="" user="" password=""
To enlarge on the details above:
- location: The path of your umbrella ISO directory, which once defined you will probably not alter.
- release: e.g. mageia5-beta1; to be changed at each new Release as announced on the QA MailList.
- user: The current rsync username e.g. isoqa; normally constant, but may change; which is announced and defined on tha QA MailList.
- password: The rsync password, given out by private e-mail; changes announced (but not given) on tha QA MailList.
Limiting your ISO set
############################################ # To exclude directories from sync, create # # a file, dorsync.skip in the start folder # # containing the directory names only, one # # per line. # ############################################
Do as it says to exclude ISOs you do not want. 'Start folder' is the directory containing your ISOs. 'Directory names' are the full ones like:
so be *very careful* when editing dorsync.skip. Moreover, you will have to do this at every Release change to reflect the new ISO directory names.
In a console/terminal, put yourself into the start directory
[if you do not, and you limit your ISO set with dorsync.skip, this will be ignored and you will download all those you do not already have];
Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q):
For "dump to USB" see the Pre-release_ISO_testing page, "Creating a bootable ISO USB stick->Dorsync".
r|R starts the download immediately: "Starting rsync".
The first time you synchronise an ISO it will necessarily be entirely downloaded; subsequently just updated with its altered parts.
At the end of all the updates, it summarises the local directory usage:
7.0G Total 3.9G Classic64 4.0K DualDVD 1.6G KDE64 1.5G Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD 136K docs 4.0K images
then checksums (if possible) all the *.iso files it finds in these directories:
Starting checks, please ensure the dates are correct PATH: Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD ISO: Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso DATE: Wed Nov 5 23:00:00 CET 2014 MD5: OK SHA1: OK PATH: KDE64 ISO: Mageia-5-beta1-LiveDVD-KDE4-x86_64-DVD.iso DATE: Sat Nov 1 00:20:00 CET 2014 MD5: OK SHA1: OK
etc, to finish if all is well:
All tests passed OK. Yippee! Please check the dates are correct.
or detail any errors found, for example:
Warning: Mageia-5-beta1-LiveCD-GNOME-en-i586-CD/Mageia-5-beta1-LiveCD-GNOME-en-i586-CD.iso.md5 missing Mageia-5-beta1-LiveCD-GNOME-en-i586-CD/Mageia-5-beta1-LiveCD-GNOME-en-i586-CD.iso.sha1 missing There are 2 potential problems.
Then it offers to make bootable ISO USB sticks:
Do you want to dump one onto USB? [y/n]
See the Pre-release_ISO_testing page, "Creating a bootable ISO USB stick->Dorsync" for the sequel.
Before using dorsync for the new Release, you must:
- Update 'release' at the start of the script
- Update accordingly all the ISO directory names and all their subordinate filenames in your local repertoire. This is quickly done in your ISOs umbrella directory by a couple of precise rename commands (and they have to be exactly right) given out on the mailing list. You can copy/paste them into a console.
- If you use the dorsync.skip file to exclude ISOs, update that also. As an aid, to list the new ISO names, do
$ RSYNC_PASSWORD="<password>" rsync --list-only rsync://email@example.com/isos/<Release>/
where <Release> is e.g. mageia5-beta1.
This command line ISO synchronisation script is quite different from the previous tools. It is aimed at people who, while they may *test* several ISOs, find synchronising one at a time quite enough - because of a slow Internet connection, for example. (Mageiasync is great for this too). It demands almost no user pre-definition or data input, working dynamically & interactively with what it finds on the server. Broadly, for the preceding tools you define the rsync remote source & local destination directories, which they subsequently use for speedy action. Onesync has no such memory; you give it the local destination ISO directory as its parameter, then steer it to the equivalent remote source ISO.
It automatically accommodates new Releases.
Onesync operates 'within' the ISO directory in question (rather than 'above' it as do the other tools), so these can have any name you like: human readable, and constant, e.g. Gnome32DVD, Classic64, KDE64CD. You can benefit from command line tab completion (quasi impossible with the full long names) whenever you look at your ISOs using a terminal/console.
The individual ISO subsidiary filenames are the full evolving ones as on the server.
It is verbose and cautious, outputting a lot of information about both ends and seeking confirmation that you are where, and seeing what, you expect.
- Normally only the rsync password to maintain.
- Name your local ISOs as you like.
- Synchronises only one at a time.
- Tests the rsync password.
- Shows both the local and remote ISO directory contents.
- Automatically reveals new Releases.
- Automatically (after confirmation) renames local ISO files accordingly.
- Checksums the *.iso image file.
It is downloadable from ???????????????? Put it where you want (ideally in a directory in you PATH) and make it executable with
$ chmod a+x onesync
# DEFINE THIS AS NECESSARY & WHEN IT CHANGES RSYNC_PASSWORD="<current password>" export RSYNC_PASSWORD # THE FOLLOWING ITEMS *COULD* CHANGE but are normally constant # rsync user USER="isoqa" # ISOs host/source directory SOURCE="bcd.mageia.org/isos"
You *must* define RSYNC_PASSWORD appropriately - by editing the script. That normally is all.
# onesync -h | DIR # -h Ouput command help # DIR The local ISO directory to synchronise # . for the current directory # If it does not exist, offer to create it
$ onesync <ISOdirectory>
where the ISO directory is either the path to it, or '.' if issued from within it. If it does not already exist, it will be (optionally) created.
The ISO you want to synchronise is confirmed:
You should now be in the local ISO directory you want to update: /mnt/common/Mageia/Gnome64 DATE.txt Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.md5 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.sha1 etc If this is correct, Enter to continue; anything else to quit:
Then you choose the release (you may see one before it is announced!):
1) mageia4-final 3) mageia5-alpha1 5) mageia5-beta1 2) mageia4-v2 4) mageia5-alpha2 #? 5
Then it lists the ISOs available for that release for you to select from:
Chosen release is: mageia5-beta1 These are its current ISOs; choose CAREFULLY: 1) Mageia-5-beta1-LiveCD-GNOME-en-i586-CD etc 4) Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD etc 9) Mageia-5-beta1-x86_64-DVD #? 4
Then it lists the remote ISO directory so that you can see if it looks sensible, spot any obvious anomaly, like a suspiciously small *.iso, or a file with an inconsistent name or date (all of which have happened):
Chosen ISO is: Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD Its current content: 29 2014/11/05 DATE.txt 1,586,495,488 2014/11/05 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso 78 2014/11/05 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.md5 etc Does it look OK? Enter to confirm, anything else to quit:
Then it checks filenames local v server, in case (after a new Release) it has to update them locally. If any are found different - it would normally be all or none -, it asks for your confirmation to rename them locally. In the event of this happening, the local directory content is re-shown to reflect changes made. This same check tells you which existing files will be updated, which missing ones downloaded.
Once launched, as for all the tools rsync included, synchronisations can take many hours. If you abandon it midway (Ctrl/C) and re-run it later, it will catch up quickly to where it left off.
When the Release changes
Nothing for you to change. The new Release will show up automatically for you to select. When onesync spots that filenames have changed, it will - with your confirmation - rename all the local ISOs files to be the same as their corresponding ones on the server. [Note that this is a blind transformation, with no knowledge of the complex filename format other than their extensions]. You will be shown the revised local ISO directory contents for information, before the synchronisation.
Rsync is extremely clever, but difficult to drive to get the results you want. One '/', '.' '*' or command option can change the outcome radically. Extracts from its man page:
rsync [OPTION...] SRC... [DEST] rsync -avz foo:src/bar /data/tmp This would recursively transfer all files from the directory src/bar on the machine foo into the /data/tmp/bar directory on the local machine. [i.e. it does *not* synchronise the contents of src/bar with /data/tmp but copies the source directory into the destination one] ... rsync -avz foo:src/bar/ /data/tmp A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name", ... In other words, each of the following commands copies the files in the same way, ... rsync -av /src/foo /dest rsync -av /src/foo/ /dest/foo [There are other basic ways of achieving the same thing, with /* or option-r]
Remember that all usage of 'rsync' requires that directory and filenames must be identical at both ends - unless you do not yet have the ISO locally.
Here are some generic example commands for different situations.
- To list the current Release directories:
RSYNC_PASSWORD="<whatever>" rsync --list-only rsync://firstname.lastname@example.org/isos/
- To list ISO directories within a Release e.g. mageia4-alpha1
RSYNC_PASSWORD="<whatever>" rsync --list-only rsync://email@example.com/isos/<release>/
- To list files within an ISO directory e.g. Mageia-5-beta1-LiveDVD-KDE4-i586-DVD
RSYNC_PASSWORD="<whatever> rsync --list-only rsync://firstname.lastname@example.org/isos/<Release>/<the_iso>/
You may want to use these in order (at least the first two) to paste the output into another command.
- To synchronise *all* the ISOs from the server into your local ISOs directory:
RSYNC_PASSWORD="<whatever> rsync -avHP rsync://email@example.com/isos/<Release>/ path/to/ISOsDir
which says "copy the contents [all ISOs] of the source Release directory into your local ISOs directory".
Or if you are in your ISOs umbrella directory:
RSYNC_PASSWORD="<whatever> rsync -avHP rsync://firstname.lastname@example.org/isos/<Release>/ .
Note the trailing source '/' [= contents of] and final '.' [= this directory].
- To synchronise one ISO directory:
RSYNC_PASSWORD="<whatever> rsync -avHP rsync://email@example.com/isos/<Release>/<the_iso> path/to/ISOsDir
which says "copy the source ISO directory itself into the destination (umbrella) one".
Or change into the local individual ISO directory and do:
RSYNC_PASSWORD="<whatever> rsync -avHP rsync://firstname.lastname@example.org/isos/<Release>/<the_iso>/ .
which says "copy the the contents of the source ISO directory into this one". The trailing slash [= contents of] and dot [= this directory] are important in this specific usage of rsync.
Finally, if you want to fix the fact that rsync disregards your UMASK setting, and if you want directories to be set with permissions 700 and files with 600, you may want to add this option, between "-avHP" and "rsync://isoqa...." :
Enough? Back to the other tools.