| Synopsis: How to test storage speed. Focus is to test removable devices, mainly "USB sticks" for use with Mageia persistent Live systems or Live or classic install to removable device. For results see Storage speed test results. |
Contents
Introduction
Depending on what you want to use it for, different parameters of a removable storage are at focus.
Sellers often only state peak read and write numbers. If you want to store much quickly you want it to have high *sustained* write speed. But then there is this other aspect of small chunks. Reading small chunks of data give lower bandwidth than reading one continuous stream. Then there is the hardest work of all for a flash memory unit: many small writes. And this is where sticks can differ by a factor of more than hundred times!
Many small writes is exactly what a normal operating system do. Especially when removing/updating/installing packages, see our use case of persistent Live system, Choosing storage. Therefore it is good to know what device to select for that purpose!
Good write up about flash devices inner workings relevant to our use: https://lwn.net/Articles/428584/
Available tools
In Mageia we have tools for testing the speed of fixed and removable media readily packaged.
KDiskMark
KDiskMark is new graphical tool available in Mageia 8. It uses Fio to perform the tests.
How to use
See at Linuxhint or Linuxuprising.
Note that it operates on the filesystem level, you tell which folder to perform the tests in. So for removable media you need to mount it and have write access.
Fio
Fio is the actual tool KDiskMark uses. It can take commands from a file.
How to use
Install our package fio, find README and example files below /usr/share/doc/fio/, open a terminal and and issue fio.
Flashbench
Flashbench is good for testing detais like erase block size on elder style flash.
How to use
Install our package flashbench, read /usr/share/doc/flashbench/README, open a terminal and and issue flashbench --help.
Test results
Moved out to Storage speed test results.
Creating and managing partitions and filesystems
Partitioning tools
Available in Mageia.
Graphical interface
gparted - Hands down my favourite.
partitionmanager - KDE Partition Manager - with link to handbook.
gnome-disks - How to use gnome-disks.
testdisk - Check and undelete partitions.
Command line
Using command line tools, more filesystems are available, and you can optimise or adapt it.
How to do it is beyond the scope of this document, but check this for a start.
Filesystems
Mageia as well as Linux systems in general supports a lot of file systems.
Some filesystems like Btrfs, Nilfs, F2fs need some more packages, which you find by name in our repos.
Comparison of a few filesystems
For more information consult Wikipedia file system list and comparison pages, and in general internet search engines for more information.
Note that if you choose other filesystems than the most common like ext4, that partition may not be readable on another system until it have relevant packages installed, such as for Btrfs, NILFS2, F2FS, ExFAT, XFS,...
Ext4 Ext4 is most standard in Unix world including Mageia, so very reliable in itself. Suggested for all normal use when you want to keep safe. Frequently used also in Android. Journal and file system check time improved over ext3, and it supports extents so it packs small files better. Lots of tools, including recovery of lost files using extundelete, ext4magic, photorec (which we all package) are available. For slow flash devices it may be interesting to gain speed by disabling the journal - If the then lower reliability is OK. Web search "disable ext4 journaling".
Ext3: In short like Ext4 without extents. Typically never preferred.
Ext2: No journal, so much less reliable in case of disruptions. Not using log is faster, but ext4 without journal is faster than ext2.
Btrfs A rather new kid on the block with lots of functionality.
NILFS/NILFS2 Optimised for flash media. Speciality: low latency. log-structured, continuous snapshots available. Seem perfect for control systems, probably F2FS is better for desktop use.
F2FS Optimised for flash media. Mageia 8 and later can use it for system partition. Used in several smartphones. Speciality: probably faster then NILFS2 in most cases. Continuous cleaning.
Microsoft compatibles
Comparisons: https://www.ntfs.com/ntfs_vs_fat.htm
NTFS The default filesystem for Microsoft operating systems since a couple decades. Journalled, so more resilient, but slower, than the FAT filesystems. In case you need to check consistency of this filesystem, a modern Microsoft OS is the best choice.
ExFAT Comes as default on removable media larger than 32GB. It is optimised for flash storage. One drawback is that is is not journalled. Rather recently Linux gained support. Exfat is encumbered by patents, so many tools do not include support for managing it, but partitionmanager and gnome-disks}} do. The package exfatprogs is needed. (Gnome-disks on Mageia 7 instead need the predecessor package exfat-utils.) How to create ExFAT partition using command line.
FAT32 Common default filesystem on removable memory devices. Wider compatibility than NTFS and ExFAT. Max sizes: file 4GB, partition 32GB.
FAT16 You may need this if you exchange information with some old equipment (and also some rather new with small processors). For everything else it is not good due to size limitations.
Optimising performance
First step in optimising is deciding what you want.
- Reliability (journalling, snapshotting, recovering tools)? => ext4 ?
- If flash, speed end endurance: F2FS
- Compatibility with other operating systems? => NTFS (more reliable) or FAT32 (less reliable but faster and suit more apparatus)
For advanced usage, several file systems can be optimised for your specific use case. Like compression, encryption, journal... details are beyond the scope of this document, but for turning off ext4 journal see ext4 below. If you are interested check out Btrfs, F2FS, NILFS2, ext4.
For optimum performance on flash devices you should align partitions to erase blocks. Not easy to find info on that, you may need to test. On modern devices it probably do not matter - but please update here and give example if you find a somewhat modern device that gain performance when aligning to other than 1MB steps.
Here is an interesting video from 2015 about flash devices, including tests.
Related information to optimising a standard Installation on a SSD (Solid State Drive).
