Wiki » Pre-release ISO testing: Part 2

Synopsis:
This page is a continuation of Pre-release_ISO_testing which got too big.

That PART 1 covers from joining Mageia QA up to creating the boot media to test.
This PART 2 covers from running & installing test ISOs to recording the results.

Multibooting considerations

In a VM or on a dedicated MBR machine the boot installation can safely take over the MBR, its default behaviour.
If sharing a real MBR machine with other systems, be sure to back up the working (pre-testing) MBR somewhere on the hard disc (or even better another media), from where it can be retrieved if necessary using a rescue CD. To save the MBR (without the partition table) of disc sda, do as root:-

dd if=/dev/sda of=<path-to-filename> bs=446 count=1

To restore it (say to disc sda) using a rescue CD, you will need to first mount the partition containing the saved MBR, then do:-

dd if=<path-to-filename> of=/dev/sda bs=446 count=1

For EFI systems, after installation the 'mageia' NVRAM entry will reference the newly installed Mageia system; as will the ESP
\EFI\mageia\grubx64.efi bootloader; regardless of whatever else you have. If you have multiple Mageia systems installed, the wiki https://wiki.mageia.org/en/About_EFI_UEFI#Other_points tells how to keep them apart.

Installing an Image

Real hardware

On MBR hardware, adapt the BIOS boot sequence so that you can boot the machine from the installation DVD or USB stick as prepared. For EFI hardware, you may have to do likewise, or possibly invoke an EFI boot menu which will enable booting from an inserted bootable DVD or USB stick. Boot from same.

Virtual Machines (VMs)

To be completed...

Live v Installation

If the Mageia image offers a 'Live' option, to use without actually installing it, that should be tried first. Then install it from the installation boot menu (or Live desktop icon), and test the installed system.

Testing the installation process

Try as many variations as you can. If you can use a language other than English, choose that for testing; it will reveal untranslated items, which you should note. Most people will be using English. Try the various boot screen:

Function key options
Menu options

On each configuration screen explore all the buttons & choices offered even if you do not actually select them. Note problems, inconsistencies, faults. If an installation (Classic) offers a choice of desktops, ideally you would install it several times for just one desktop each time, then install with the whole lot for usage & application testing. This would be enormously demanding, so more realistically, stick to choosing several or all of the offered desktops, because other (Live) ISOs are desktop specific - Xfce, KDE or Gnome - so *they* will be well tested on their own.

You will find that the first installation of each type you try, Live or Classic, will involve a lot of exploration and note taking. Subsequent installations go faster because you already know the pitfalls, and all the options you actually need for your own circumstances.

If you have a multi-boot management system, you may want to adapt that to include the newly installed Mageia system before re-booting the machine after installation. If so, use Ctrl/Alt/Fn to get a console to play with as necessary; then Ctrl/Alt/F1 to get back to the finished installation screen, then re-boot as invited.

Installation guidelines: what to test and report on the PAD

Do include your media type

As learned from the pretesting, some media perform differently, or present different options during install. If you report USB media, another tester will hopefully be prompted to use DVD media.

Do include your Installation type and method:

UEFI or CSM / legacy boot?
Virtual box ( type) or real hardware?
Is it a clean install onto new partitions ( and which type : ext3/4, btrfs ....etc )?
Or onto existing partitions using Custom Disk Partitioning? (or other options from the installer)
Did you use the net installer. either from a local .iso on a HDD or online mirror?

These all present installation issues in different ways, so the greater the variety of install methods, the higher the chance of finding any issues.

Do open a bug if you find something.

Do include your installation method for the Classic.

A single DE or Multiple DE install.

Is it an Upgrade?

It has been shown that a Multi DE install can have an installation failure due to a package conflict, yet a single DE install that uses the same package will install without incident. Also, sadly, the reverse is also true.

Testing the Installed System

Once again, the first time will most arduous because you have potentially everything to explore and try, discovering what works or does not. Note all problems. On subsequent test installations you will tend to concentrate on known issues to see whether they are still extant, accepting that what worked before still does. Which does not preclude new faults introduced along the way.

Try as much as you can, as many applications as possible. Try at least *starting* every single one in the menus - some may not! If there is a desktop choice, try all desktops at least initially. Sometimes application or display problems happen on one desktop but not on another. Try all sound and video applications from local media. Try the Internet - sound & video included - with all browsers installed.

Guidelines for what to test and report on the PAD

Display manager(s)

A single DE install will also check the correct functioning of the Display Managers (sddm, GDM, LightDM, ... etc); multiple desktop installs include the different DMs, and these can be selected via MCC/Startup/Display managers. Check that the DM greeter is working correctly. If you can, check that auto-login also works.

Do check the Mageia Control Centre.

This is one of the things that set Mageia apart from other Distributions. Invoke it from both CLI and from the application launchers if you are able. There should be one on all desktop panels; and in the Mageia Menu

Check that all the modules at least launch, and if you have the skills, check their operation, e.g. network connectivity

Do check Mageia Welcome.

This is the first thing a new installation presents to the user, and is present in many Distributions. There are 30ish links to check, both internet and on-system. Check that at least one Applications can be installed (you need to set up online media first)

Mageia Application Menu.

There are a lot of launchers in here that need to be tested. Try (launch) every single one. Do they all have an icon? Also, check briefly things most users are likely to use, e.g.
- Browsers
- LibreOffice
- Sound players produce sound.
- Media players play video correctly and have sound.
- Photo, image processors display images correctly.

Finally, check the logout / reboot and shutdown dialogues work as expected.

When reporting on the above, brevity and clarity count. Other reporters are likely reluctant to read through screeds of text to find that you found that you found nothing wrong. On the other hand if you do find something wrong, a brief line with a bug report # is great. e.g.

Plasma desktop shows old KDE logo not current Plasma logo, #20647

says what the problem is and the bug #.

Just in case you missed it:

Do open a bug if you find something.

If it has previously been reported, triage will let you know. If you add the bug # to the pad, other testers can add confirmation to the bug.
This will aid the lucky coder assigned to fix the issue, confirmation that at least 2 or more users have met the issue on different hardware.

Managing & Testing System Upgrades

For example, Mageia 3 to Mageia 4, using the so-called Classic Installer DVDs.

Preparation

If you regularly use Mageia anyway, your normal system can be used as the basis for upgrade testing. If not, install an appropriate current Mageia downloaded from the public site: the biggest all-desktops one is most demanding, otherwise one of the Xfce, KDE or Gnome DVDs. In both cases, ensure that the system is then brought *up-to-date*: this is a normal requirement for upgrading.

It is also a good idea to remove the initial install report.bug.xz. before commencing the upgrade, or even before you make a copy of your system. If you remove these before archiving, you only need to do it the once, not every time you restore from the archive. This is so that if an installation issue occurs, there is only the one report.bug.xz. available to attach to your bug report.

Then archive it as follows.

Because you will be doing more than a single upgrade test, it is important now to *save the base Mageia system* so that it can be restored for multiple upgrade tests. VM users can simply copy the reference VM. Real h/w users will need to use software like PartClone which copies a disc partition to a *file* anywhere you have space for it, and restores the partition from that file. This save/restore process can only be done from another running system, as root, either on the same box or live rescue CD. Using PartClone on partition e.g. sda3 formatted as some extfs filesystem, the command to save initially the reference system [guessed ? ]:

partclone_extfs -a -s /dev/sda3 -o <path-to-filename>

Prior to each re-test of the upgrade (no need the first time), you must re-set the reference system partition e.g. sda3 to the 'old' Mageia, as root again from another system:

partclone_extfs -r -s <path-to filename> -o /dev/sda3

Testing the Upgrade Process

Up to at least Mageia 3, 'upgrade' was not offered at the installation boot menu, but you had to choose 'install' then find 'Upgrade existing Mageia system' as an option later on. Hopefully this will be clearer with Mageia 4. Whatever, once the 'upgrade' option is taken, follow the process through - it is much slicker than an installation - and note any problems you see or experience.

Testing the Upgraded System

It is worth testing the resulting 'new' Mageia as far as you can, along the lines of 'Testing the installed system' above, ensuring that things continue to work as before and there are no regressions.

Recording Your Results

Each pre-release phase has its own interactive 'whiteboard', alias PAD, whose URL is given out at the time. This PAD has pre-defined headings/sections for all the ISO variants, grouped appropriately, 'live' and 'install'. Test results are written into the section relevant to the variant you tested. You always start by clicking as invited in the top right corner to log in (unless the cooky/site's scripts and user ip already recognize you) and be given a colour for *your* comments. When defining a new user/colour make it as contrasting from others as you can.
For others to recognise your work, please add your alias/name under "Testers and other users" to simplify discussion on channels, in meetings etc.

Note every fault you find for installation/upgrade/usage, and anything you consider particularly inconvenient, confusing or difficult.

If you find a definite fault which needs correction, *raise a bug* for it on Bugzilla (https://bugs.mageia.org/) unless someone has already done so. Look out for that on the test results PAD, especially it's archive page, ongoing mailing list messages, and in Bugzilla itself. Note that the codebase for ISO testing is called 'Cauldron', which is the current software development working set; Bugzilla needs this.

Usually the pre-release testing of the isos will go through an intensive and interactive process of successive builds, or writing of iso images. As the testers communicate their findings via the pad, irc, mailing list and Bugzilla to the developers, problems are being (attempted to be) fixed and the team keeps their working notes current in the pad. To be able to distinguish between the findings for the successive builds one of the responsible testers and/or developers should archive the notes for everyone's reference and wipe the data on the current pad clean upon arrival of a new build of a set of isos, so the findings for each new build can be written again onto the pad. This way an earlier finding that the testers discover has not been fixed in a newer build, can be easily copied and pasted from an archive page to a current page, so as to avoid having to describe a recurring problem anew every time it is found.

A lot of problems you find during testing are more inconveniences than faults: it works, but less smoothly than could be. Or maybe you alone seem to hit a problem: is it real? The #mageia-qa channel is the primary tool to discuss anything with fellow testers. The mailing list is useful for airing doubts you have. People will reply as to whether you should raise a bug on something or not. It is also useful for airing your general views about the installations & upgrades you try, and problems you have with procedures or information shortcomings. Do not be shy of saying what you think; it may yield improvement.

Maintaining the preliminary notepad

Suppose that the current build being tested is M5B1, the notepad also referred to as 'riseup' pad would be named like: https://pad.riseup.net/p/mageia5beta1 and each new build will keep the notes in a related archival page, in our example called: https://pad.riseup.net/p/mageia5beta1-archive

The procedure for archiving is simple:

1. Go to the URL for the archive (by changing the address you create it, or arrive there if it is existing).

2. Copy & paste all data for the isos that are being replaced by a new build and write them down into the archive. This will remove all colour codes and paste in your own colour. Your own is not applicable, so select the new paste and discolour it (with the coloured circular icon in the upper left).

3. Please paste each build's data on top of the old, separate them by a divide to show which is what. So newest is top of the page.

Hereafter, if you find the same remarks in build X and build X+1, you can easily copy and paste such remarks back to the current build data.

4. Now the regular pad is ready for the new build, and all written data can be replaced. But in order that one does not have to clean each variable in every line, a blank template of this pad is available here: https://wiki.mageia.org/en/File:M5b1-pad-template-v.20141030-1932.txt. But probably easier to handle (as long as it lasts) is this one: https://pad.riseup.net/p/mageia-qaiso-template-keep. According to the riseup info it stays in place a year after the last change.

Things to try when things go wrong

This section suggests console commands which you can usefully employ in the different circumstances, and specific files to look at or capture, to enable diagnosis of the problem.

The Installation/Upgrade Medium Does Not Boot

If possible, try it on another machine. See from the mailing list and PAD whether other people have the same problem. Note anything you see on the screen. If you suspect the medium, rather than simply re-writing it, it is worth checking its integrity as described below. Clearly if this check is good, the problem is real - report it. If the check shows a fault, throw the DVD or USB stick and write another one.

Verifying a bootable installation DVD

From your working system, note the file size of the original ISO image.
Divide by 2048 to get the *number of blocks*, say xxxxx.
Then copy the CD/DVD back onto disc (assuming the optical drive is /dev/sr0):-

dd if=/dev/sr0 of=<destination filename> bs=2048 count=xxxxx

If the command shows a wrong total byte count, the medium *was* faulty; good to know. If the byte count is OK, checksum the file (e.g. md5sum <path-to-ISO-copy.iso>) and verify the result against the original ISO's given checksum *.md5.

Verifying a bootable ISO USB stick

To be completed...

Some kernel options to try if you can edit the boot command line

it might be a graphics card/driver issue. Try adding the kernel option xdriver=vesa .
another thing to do: remove "splash silent" from command line so you can see exactly where it stalls
there is still some timing issues, so you could also try to add "divider=10" [this applies particularly to VirtualBox installations, and apparently some elderly i586 machines].
also you can try to remove the "vga=788" part too in case it's our bootsplash sucks again

Sometimes the boot process fails and issues an error report but falls back to a debug shell which allows the user to investigate further or at least save a debug file to an external medium like a USB pen drive. This shell provides access to a few useful commands such as 'ls', 'mount' and 'cp'.

A case in point:

The boot process exits the graphic screen and presents a command line after a series of error messages starting with "hwdb.bin does not exist". It might help the developers and packagers to have a copy of the file rdsosreport.txt which the system writes into RAM at some stage.

Plug in a USB stick and use "ls dev/sd*" to see which mass storage devices are attached and identify your stick. Pick the latest in the list: e.g. sdc1 if it contains sda, sdb, sdc plus their partitions, then create a mount point and mount the device there;

  mkdir <whatever>
  mount -t ext4 /dev/sdc1 <whatever>

Your file system type is more likely to be DOS or FAT32 in which case replace 'ext4' with 'vfat'.

  cp run/initramfs/rdsosreport.txt <whatever>/....   
  umount <whatever>

The Installation Does Not Complete

Much depends here how soon it fails, whether anything has yet been written to disc, whether you can access a working console with Ctrl/Alt/Fn, whether the system is frozen. If the installation fails randomly, especially during the 'install to disc' phase, suspect the medium and verify it as per "Verifying a bootable ..." above.

For classical isos

If you can switch to tty2 with Ctrl/Alt/F2:

insert a USB key
type bug
when the system is ready, remove the installation medium and reboot with Ctrl/Alt/Del
compress report.bug that was written to the USB key with the command xz report.bug
attach report.bug.xz to your bug report and assign to tv

The Installation/Upgrade Completes But The Subsequent Re-Boot Fails

Again what you can do depends on how far things got: whether the system seizes, whether you can get a working console etc. The section above "Some kernel options to try if you can edit the boot command line" may advance things.

For classical isos:

Fetch /root/drakx/report.bug.xz or /root/drakx/report.bug
- In case you upgraded: look at the date, to make sure you fetch the correct file
- use a LiveCD/DVD or a Mageia install on another partition if you can't log into your newly installed Mageia at all.
Compress it if it isn't already compressed
- To compress type xz report.bug in a terminal (after going to the correct directory).
Attach report.bug.xz to your bug report and assign to tv

Reboots To A Console Rather Than Graphical Login Screen

First check whether the graphical login screen is on a different tty, by pressing ctrl+alt+Fn (you can try all twelve of them, but it is most probably on F1 or F2)

If that doesn't help: Regardless of whether you ended up in rescue mode or in multi-user mode, you can still fetch useful logs.

In rescue mode, you'll be told that you need to enter your root password. In multi-user mode you can log in the normal way, with user name (this time use "root") + password.

Now type:
journalctl -b > journalctl-b.txt
and then do one of the following:

(TODO: add how to mount a usb stick and then copy this file to it, after figuring out how to do that)
later fetch this partition's /root/journalctl-b.txt using a working partition or a Live iso.

if you're not in rescue mode, but had to enter a user name + password, you can also try, as root:
systemctl start dm.service
to see whether that brings the login screen. Again, it can come on a different tty (different ctrl+alt+Fn) than where you are now.

If that file exists, it is useful to fetch /var/log/Xorg.*.log* from the failed boot, too (look at time + date from the file to see whether you need Xorg.0.log or a different one)