From Mageia wiki
Revision as of 08:37, 11 June 2020 by Cmoifp (talk | contribs)
Jump to: navigation, search


Drakconf multiflag.png
Other languages
English; Francais;


Synopsis:
This page describes tools to facilitate the management of Mageia ISOs for pre-release testing. It also details the use of rsync directly for this purpose. It is a supplement to the "Pre-release ISO testing" page.

Introduction

Mageia ISOs are hosted on a server: bcd.mageia.org/isos.

Here there is a directory for each Release e.g.

mageia4-final
mageia4-v2
mageia5-alpha1
mageia5-alpha2
mageia5-beta1

Each Release comprises all the ISOs, e.g. for mageia5-beta1:

Mageia-5-beta1-LiveCD-GNOME-en-i586-CD
Mageia-5-beta1-LiveCD-KDE4-en-i586-CD
Mageia-5-beta1-LiveDVD-GNOME-i586-DVD
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD
Mageia-5-beta1-LiveDVD-KDE4-i586-DVD
Mageia-5-beta1-LiveDVD-KDE4-x86_64-DVD
Mageia-5-beta1-dual-DVD
Mageia-5-beta1-i586-DVD
Mageia-5-beta1-x86_64-DVD

In fact each so-called ISO is a directory which contains not only the actual ISO image file *.iso, but additional files like date, checksums. For example (the Classic DVD):-

DATE.txt
Mageia-5-beta1-x86_64-DVD.idx
Mageia-5-beta1-x86_64-DVD.iso
Mageia-5-beta1-x86_64-DVD.iso.md5
Mageia-5-beta1-x86_64-DVD.iso.sha1

or (a Live ISO):-

DATE.txt
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.md5
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.sha1
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.langs
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst.full
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst.leaves
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.lst.names

Note well that both the ISO directory and subordinate file names incorporate the Release. This has consequences for you depending on which rsync tool you use.

The ISOs you are testing reside in a local directory of your choice. To keep them synchronised with those on the server, these ISO directory (mostly) and subordinate file names must be identical. Those on the server change at each new release (in effect, the download source changes). How you keep your local copies in line with those on the server is described for each of the tools - some do it for you, others you have to do it yourself.

Why have tools?

Both the initial download (priming) & subsequent updating of your local ISOs are done using rsync, which has the powerful & important ability to download only the changed parts of files, noteably the large ISO images, greatly reducing download times. The use of rsync is very exact: it requires great care to make it do just what you want, and the long ISO directory names make difficult parameters. If you get it wrong, you will probably download entire ISOs you do not want. Its specific usage is covered later in this page. The tools described beforehand will make your life much easier.

Global rsync information

Any use of rsync in our context includes certain basic parameters, which are required by all the methods described, and defined for them in different ways:

  • The main ISO source, the server: bcd.mageia.org/isos. This is permanent.
  • An rsync 'user': e.g. isoqa. This remains constant over the long term, but *can* be changed, in which case it is announced on the QA MailList.
  • An rsync password: this is given out only by private e-mail, initially and when it periodically changes.

Applies to all the tools: despite its amazing capabilities, it seems that any line break stops rsync: it does not recover/resume after the break. You probably will have to abort it. Re-running the synchronisation, it will quickly catch up to where it got to, and carry on. Thus, if necessary you can interrupt an ongoing transfer without significant penalty when you re-do later.

MageiaSync

mageiaSync is a tool to get one or more ISOs images of Mageia, from testing repositories or from mirrors. It is the main tool for ISO testers, the only one with a GUI. It manipulates ISOs under their full long (and evolving) names. It has a facility (after a new Release) to rename local ISO directories & files - on demand.

Functionalities

  • GUI
  • Lists remote and local ISO directories and image files.
  • Allows the selection of one or more images to download/update.
  • Verifiy the signatures of MD5 and SHA512 sums provided and performs checks of them.
  • Provides a tool for Release changes, i.e. renaming directories and files with the new name.

Installation

An RPM package mageiasync is available for Mageia 5 and later.

Usage

To launch it, there is a menu entry "MageiaSync" under System Tools. From a console, simply type:

$ mageiasync

First launch

At first launch, you have to give some information. A Preferences dialogue asks for:
MageiaSync2.png

  • User: this is the rsync username e.g. isoqa .
  • Password : the current rsync password.
  • Source: The download ISOs source. Normally leave this empty for the main Mageia ISO source; otherwise give a mirror address with the path to the release.

With this information, the main Preferences dialogue appears:
MageiaSync1.png

  • Release: The drop-down list shows all the releases currently available, e.g. mageia5-alpha2. Choose the one that interests you, normally the latest.
  • User [as above]
  • Password [as above]
  • Source [as above]
  • Destination: your local directory where the ISOs will be stored. Click it to raise a file picker dialogue [beware: this may be hidden beneath other windows].
  • Bandwidth limit: keep 0 if you don't want to apply limit.

Menu File->Preferences enables you to edit these values subsequently.
[Config file is ~/.config/Mageia/mageiaSync.conf]

If the top 'Release' frame is void, then some information are wrong, use File->Preferences to adjust them, then File->Update list.
Clicking OK leads to the main dialogue below.

Regular use: the main window

This appears directly for any subsequent use of Mageiasync, using the pre-defined information to display immediately all that is required to 'synchronise' your current ISO set.
MageiaSync.png

  • The Remote directory frame lists all the ISOs at the server for the current release. Select the one(s) you want. If you do not already have it, it will be downloaded; otherwise it will be updated. The Select All button does what it says - if you really want the whole set.
  • the Local directory frame identifies your local ISO umbrella directory, and lists the ISOs you currently have in it.
  • The Do sync button launches the synchronisation. This can take hours, but if resumed later recovers very efficiently; so do not be frightened of using the Stop button if you need to.

After the download completion, the signatures of MD5 and SHA512 sums stored locally are verified, and the sums are computed and checked versus the ones stored in files. If signature files are provided, the signature is verified and the sums are matching, 'OK' will be displayed. If 'Failed' is displayed, thus one of the previous conditions is not true. See this example when MD5 sum is signed, with the icon showing that the signature is OK. Mageiasync5.png

When the Release changes

At a new release, all your local ISO directory and subordinate filenames must be updated to be the same as their new equivalents on the server. From the menus, Sync-> Rename archives shows the Rename release dialogue.
MageiaSync3.png

Warning!
The current and new release names used here are as they exist in directory & file names (e.g. Mageia-5-alpha2), *not* as the Release title (e.g. mageia5-alpha2). Be very careful what you type.


  • Base directory: shows your current ISO umbrella directory; click if you want to change it (unlikely: you would end up downloading entire ISOs).
  • Old release: e.g. Mageia-5-alpha2
  • New release: e.g. Mageia-5-beta1

OK and it is done! You will have the number of directories and files affected by the change.

Dorsync

This command line shell script was the main tool before the advent of Mageiasync. It originally assumed that you juggled *all* the ISOs, but now allows the exclusion of those you do not want. It manipulates ISOs under their full long (and evolving) names. Apart from the main business of synchronising your ISOs, it will make bootable ISO USB sticks of them if you want - the only rsync tool to do so.

Features

  • Operates implicitly on the entire ISO set, but ...
  • Allows specific ISOs to be excluded.
  • Checksums the *.iso image files.
  • Can create bootable ISO USB sticks both after synchronisation, and independantly.

Installation

dorsync can be easily downloaded via

$ wget http://gitweb.mageia.org/qa/dorsync/tree/dorsync

Put it where you want (ideally in a directory in you PATH) and make it executable with:

$ chmod a+x dorsync

Usage

Preparation

Before you use dorsync, you must define, by editing the start of it, several values:

############################################
# Enter a few variables.                   #
# location is where you stored the ISOs    #
# release is the directory on the server   #
# user is the rsync login username         #
# password is the rsync password           #
############################################

location=""
release=""
user=""
password=""

To enlarge on the details above:

  • location: The path of your umbrella ISO directory, which once defined you will probably not alter.
  • release: e.g. mageia5-beta1; to be changed at each new Release as announced on the QA MailList.
  • user: The current rsync username e.g. isoqa; normally constant, but may change; which is announced and defined on tha QA MailList.
  • password: The rsync password, given out by private e-mail; changes announced (but not given) on tha QA MailList.

Limiting your ISO set

############################################
# To exclude directories from sync, create #
# a file, dorsync.skip in the start folder #
# containing the directory names only, one #
# per line.                                #
############################################

Do as it says to exclude ISOs you do not want. 'Start folder' is the directory containing your ISOs. 'Directory names' are the full ones like:
Mageia-5-beta1-LiveCD-GNOME-en-i586-CD
Mageia-5-beta1-x86_64-DVD
so be *very careful* when editing dorsync.skip. Moreover, you will have to do this at every Release change to reflect the new ISO directory names.

Usage

In a console/terminal, put yourself into the start directory
[if you do not, and you limit your ISO set with dorsync.skip, this will be ignored and you will download all those you do not already have];
just type:

$ dorsync

It starts

Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q):

For "dump to USB" see the Pre-release_ISO_testing page, "Creating a bootable ISO USB stick->Dorsync".
r|R starts the download immediately: "Starting rsync".
The first time you synchronise an ISO it will necessarily be entirely downloaded; subsequently just updated with its altered parts.
At the end of all the updates, it summarises the local directory usage:

7.0G	Total
3.9G	Classic64
4.0K	DualDVD
1.6G	KDE64
1.5G	Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD
136K	docs
4.0K	images 

then checksums (if possible) all the *.iso files it finds in these directories:

Starting checks, please ensure the dates are correct

PATH: Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD
ISO:  Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso
DATE: Wed Nov  5 23:00:00 CET 2014
MD5:  OK     SHA1: OK

PATH: KDE64
ISO:  Mageia-5-beta1-LiveDVD-KDE4-x86_64-DVD.iso
DATE: Sat Nov  1 00:20:00 CET 2014
MD5:  OK     SHA1: OK 

etc, to finish if all is well:

All tests passed OK. Yippee!
Please check the dates are correct.

or detail any errors found, for example:

Warning:
Mageia-5-beta1-LiveCD-GNOME-en-i586-CD/Mageia-5-beta1-LiveCD-GNOME-en-i586-CD.iso.md5 missing
Mageia-5-beta1-LiveCD-GNOME-en-i586-CD/Mageia-5-beta1-LiveCD-GNOME-en-i586-CD.iso.sha1 missing
There are 2 potential problems.

Then it offers to make bootable ISO USB sticks:

Do you want to dump one onto USB? [y/n]

See the Pre-release_ISO_testing page, "Creating a bootable ISO USB stick->Dorsync" for the sequel.

Release changes

Before using dorsync for the new Release, you must:

  • Update 'release' at the start of the script
  • Update accordingly all the ISO directory names and all their subordinate filenames in your local repertoire. This is quickly done in your ISOs umbrella directory by a couple of precise rename commands (and they have to be exactly right) given out on the mailing list. You can copy/paste them into a console.
  • If you use the dorsync.skip file to exclude ISOs, update that also. As an aid, to list the new ISO names, do
$ RSYNC_PASSWORD="<password>" rsync --list-only rsync://isoqa@bcd.mageia.org/isos/<Release>/

where <Release> is e.g. mageia5-beta1.

Onesync

This command line ISO synchronisation script is quite different from the previous tools. It is aimed at people who, while they may *test* several ISOs, find synchronising one at a time quite enough - because of a slow Internet connection, for example. (Mageiasync is great for this too). It demands almost no user pre-definition or data input, working dynamically & interactively with what it finds on the server. Broadly, for the preceding tools you define the rsync remote source & local destination directories, which they subsequently use for speedy action. Onesync has no such memory; you give it the local destination ISO directory as its parameter, then steer it to the equivalent remote source ISO.
It automatically accommodates new Releases.

Onesync operates 'within' the ISO directory in question (rather than 'above' it as do the other tools), so these can have any name you like: human readable, and constant, e.g. Gnome32DVD, Classic64, KDE64CD. You can benefit from command line tab completion (quasi impossible with the full long names) whenever you look at your ISOs using a terminal/console.
The individual ISO subsidiary filenames are the full evolving ones as on the server.

It is verbose and cautious, outputting a lot of information about both ends and seeking confirmation that you are where, and seeing what, you expect.

Features

  • Normally only the rsync password to maintain.
  • Name your local ISOs as you like.
  • Synchronises only one at a time.
  • Tests the rsync password.
  • Shows both the local and remote ISO directory contents.
  • Automatically reveals new Releases.
  • Automatically (after confirmation) renames local ISO files accordingly.
  • Checksums the *.iso image file.

Installation

It is downloadable from https://bugs.mageia.org/attachment.cgi?id=11685&action=edit Put it where you want it (ideally in a directory in you PATH), and make it executable with $ chmod a+x onesync

Changeable information

# DEFINE THIS AS NECESSARY & WHEN IT CHANGES
RSYNC_PASSWORD="<current password>"
export RSYNC_PASSWORD

# THE FOLLOWING ITEMS *COULD* CHANGE but are normally constant
# rsync user
USER="isoqa"
# ISOs host/source directory
SOURCE="bcd.mageia.org/isos" 

You *must* define RSYNC_PASSWORD appropriately - by editing the script. That normally is all.

Usage

# onesync -h | DIR
# -h    Ouput command help
# DIR   The local ISO directory to synchronise
#               . for the current directory
#               If it does not exist, offer to create it 

Simply:

$ onesync <ISOdirectory>

where the ISO directory is either the path to it, or '.' if issued from within it. If it does not already exist, it will be (optionally) created.

The ISO you want to synchronise is confirmed:

You should now be in the local ISO directory you want to update:
	 /mnt/common/Mageia/Gnome64
DATE.txt
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.md5
Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.sha1
 etc
If this is correct, Enter to continue; anything else to quit: 

Then you choose the release (you may see one before it is announced!):

1) mageia4-final   3) mageia5-alpha1  5) mageia5-beta1
2) mageia4-v2	   4) mageia5-alpha2
#? 5 

Then it lists the ISOs available for that release for you to select from:

	 Chosen release is: mageia5-beta1
These are its current ISOs; choose CAREFULLY:
1) Mageia-5-beta1-LiveCD-GNOME-en-i586-CD
 etc
4) Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD
 etc
9) Mageia-5-beta1-x86_64-DVD
#? 4 

Then it lists the remote ISO directory so that you can see if it looks sensible, spot any obvious anomaly, like a suspiciously small *.iso, or a file with an inconsistent name or date (all of which have happened):

	 Chosen ISO is: Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD
Its current content:
            29 2014/11/05 DATE.txt
 1,586,495,488 2014/11/05 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso
            78 2014/11/05 Mageia-5-beta1-LiveDVD-GNOME-x86_64-DVD.iso.md5
 etc
Does it look OK? Enter to confirm, anything else to quit: 

Then it checks filenames local v server, in case (after a new Release) it has to update them locally. If any are found different - it would normally be all or none -, it asks for your confirmation to rename them locally. In the event of this happening, the local directory content is re-shown to reflect changes made. This same check tells you which existing files will be updated, which missing ones downloaded.

Once launched, as for all the tools rsync included, synchronisations can take many hours. If you abandon it midway (Ctrl/C) and re-run it later, it will catch up quickly to where it left off.

When the Release changes

Nothing for you to change. The new Release will show up automatically for you to select. When onesync spots that filenames have changed, it will - with your confirmation - rename all the local ISOs files to be the same as their corresponding ones on the server. [Note that this is a blind transformation, with no knowledge of the complex filename format other than their extensions]. You will be shown the revised local ISO directory contents for information, before the synchronisation.

Rsync directly

Rsync is extremely clever, but difficult to drive to get the results you want. One '/', '.' '*' or command option can change the outcome radically. Extracts from its man page:

              rsync [OPTION...] SRC... [DEST]

              rsync -avz foo:src/bar /data/tmp
       This would recursively transfer all files from the directory src/bar on
       the machine foo into the /data/tmp/bar directory on the local  machine.
       [i.e. it does *not* synchronise the contents of src/bar with /data/tmp
        but copies the source directory into the destination one]
       ...
              rsync -avz foo:src/bar/ /data/tmp
       A trailing slash on the source changes this behavior to avoid  creating
       an  additional  directory level at the destination.  You can think of a
       trailing / on a source as meaning "copy the contents of this directory"
       as  opposed  to  "copy  the  directory  by name",
       ...
       In other words, each of the following
       commands copies the files in the same way, ...
              rsync -av /src/foo /dest
              rsync -av /src/foo/ /dest/foo
       [There are other basic ways of achieving the same thing, with /* or option-r]

Remember that all usage of 'rsync' requires that directory and filenames must be identical at both ends - unless you do not yet have the ISO locally.

Here are some generic example commands for different situations.

  • To list the current Release directories:
RSYNC_PASSWORD="<whatever>" rsync --list-only rsync://isoqa@bcd.mageia.org/isos/
  • To list ISO directories within a Release e.g. mageia4-alpha1
RSYNC_PASSWORD="<whatever>" rsync --list-only rsync://isoqa@bcd.mageia.org/isos/<release>/
  • To list files within an ISO directory e.g. Mageia-5-beta1-LiveDVD-KDE4-i586-DVD
RSYNC_PASSWORD="<whatever> rsync --list-only rsync://isoqa@bcd.mageia.org/isos/<Release>/<the_iso>/

You may want to use these in order (at least the first two) to paste the output into another command.

  • To synchronise *all* the ISOs from the server into your local ISOs directory:
RSYNC_PASSWORD="<whatever> rsync -avHP rsync://isoqa@bcd.mageia.org/isos/<Release>/    path/to/ISOsDir

which says "copy the contents [all ISOs] of the source Release directory into your local ISOs directory".
Or if you are in your ISOs umbrella directory:

RSYNC_PASSWORD="<whatever> rsync -avHP rsync://isoqa@bcd.mageia.org/isos/<Release>/  .

Note the trailing source '/' [= contents of] and final '.' [= this directory].

  • To synchronise one ISO directory:
RSYNC_PASSWORD="<whatever> rsync -avHP rsync://isoqa@bcd.mageia.org/isos/<Release>/<the_iso>    path/to/ISOsDir

which says "copy the source ISO directory itself into the destination (umbrella) one".
Or change into the local individual ISO directory and do:

RSYNC_PASSWORD="<whatever> rsync -avHP rsync://isoqa@bcd.mageia.org/isos/<Release>/<the_iso>/  .

which says "copy the the contents of the source ISO directory into this one". The trailing slash [= contents of] and dot [= this directory] are important in this specific usage of rsync.

Finally, if you want to fix the fact that rsync disregards your UMASK setting, and if you want directories to be set with permissions 700 and files with 600, you may want to add this option, between "-avHP" and "rsync://isoqa...." :

   --no-p --chmod=D700,F600

Enough? Back to the other tools.