Where do security updates come from?

by David Walser

Abstract

One common piece of guidance to computer users is to keep your operating system patches and updates up to date, but what is seldom discussed is how those patches are getting created. I will speak from the standpoint of someone on a Linux distribution security team about the process behind creating and issuing those security updates. The processes goes from gathering information about vulnerabilities, assessing their relevance and importance, deciding how and when to handle them, creating updated or patched packages, and working with the QA team to get them tested and released.

Introduction

I have been the leader of the Packages Security Team [1] for Mageia since 2012 [2]. Mageia [3] (formerly known as Mandriva Linux, and before that, Linux Mandrake) is one of the most popular Linux distributions [4]. I have been contributing to it since 2001 and using it since 1999. The primary responsibility of the Packages Security Team is collect information about security vulnerabilities that affect packages that are included in the Mageia Linux distribution, make sure that bugs are filed for these issues and include the appropriate needed information, communicate with developers and packagers as needed to ensure that fixed packages get built, and work with the QA team who tests the updated packages to ensure that they are ready to be released as updates.

Steps in the life-cycle of a security update

Becoming aware of an issue
Determining its applicability to our products
Locating a fix
Implementing a fix and building the update
Writing an advisory for the update
Testing (QA)
Releasing the update

Becoming aware of an issue

Linux distributions contain code from many independent developers all over the world. Security vulnerabilities are found and disclosed by many different entities, including upstream developers, vendors, and security researchers. It is impossible to track all the initial sources of information. Knowing good sources of information about publicly disclosed security vulnerabilities and communication within the open-source and security communities is key. There are four primary sources of information that Linux distribution security teams use to become aware of security issues that potentially affect their products:

Mailing lists
Other vendor advisories
Upstream changelogs and release announcements
Reports from users

Mailing lists

There are mailing lists [5] that are used primarily to exchange information about security vulnerabilities affecting open-source software that we can follow to become aware of issues. The primary list among these is the oss-security [6] mailing list, where discussion of publicly-disclosed security vulnerabilities in open-source software takes place. For many issues, the initial public disclosures are made on this list. There are also private mailing lists for discussion of issues that have not yet been made public (i.e., issues that are under embargo) but will be publicly disclosed soon. These mailing lists, linux-distros and distros, only have members of open-source operating system security teams as members, with the difference between them being that linux-distros only includes Linux distributors and distros also includes BSD UNIXes. As of this writing, Mageia's security team is not subscribed to these lists. The FullDisclosure [7] mailing list contains information about security vulnerabilities for which fixes may not yet be available, and is therefore less useful in the process of packaging security updates, but may be a useful resource in some circumstances nonetheless. Finally, the Bugtraq [8] list collects security advisories from various vendors and upstream developers.

Other vendor advisories

Linux distribution security teams become aware of security issues many different ways, and other than possibly creating an entry in a bug tracker or security issue tracker (access to which may initially be restricted to only their security team, especially for higher-impact issues), they generally do not broadcast this information until they have a fix released and they issue a security advisory. For the majority of security issues that I become aware of, it happens by reading these security advisories from other Linux distributions when they issue their updates. Fortunately, the Linux Weekly News website has a vulnerabilities database [9] where they collect these advisories and index them by the security issues fixed in those updates. This allows me to have one location from which I can track virtually all of the issues that I need to be concerned with, with links to the advisories and names of the packages that were fixed in each advisory all listed in one place. I can use this to see issues that we have not yet fixed, as well as to determine when a particular vulnerability affects more than one package.

csync2: file checksum collision
Package(s):	csync2 	CVE #(s):	CVE-2014-8242
Created:	March 20, 2015 	Updated:	October 15, 2015
Description: 	

From the Fedora advisory:

librsync previously used a truncated MD4 "strong" check sum to match blocks. However, 
MD4 is not cryptographically strong. It is possible that an attacker who can control the 
contents of one part of a file could use it to control other regions of the file, if it is 
transferred using librsync/rdiff. For example this might occur in a database, mailbox, or 
VM image containing some attacker-controlled data. To mitigate this issue, signatures will 
by default be computed with a 256-bit BLAKE2 hash. Old versions of librsync will complain 
about a bad magic number when given these signature files. Backward compatibility can be 
obtained using the new `rdiff sig --hash=md4` option or through specifying the "signature 
magic" in the API, but this should not be used when either the old or new file contain 
untrusted data. Deltas generated from those signatures will also use BLAKE2 during 
generation, but produce output that can be read by old versions.

Alerts: 	
openSUSE 	openSUSE-SU-2015:1752-1 rsync   	2015-10-15
Mandriva 	MDVSA-2015:204 	        librsync 	2015-04-27
Mageia 	        MGASA-2015-0146 	librsync 	2015-04-15
Arch Linux 	ASA-201503-10 	        librsync 	2015-03-16
Fedora 	        FEDORA-2015-3497 	rdiff-backup 	2015-03-19
Fedora 	        FEDORA-2015-3366 	rdiff-backup 	2015-03-19
Fedora 	        FEDORA-2015-3497 	librsync 	2015-03-19
Fedora 	        FEDORA-2015-3366 	librsync 	2015-03-19
Fedora 	        FEDORA-2015-3497 	duplicity 	2015-03-19
Fedora 	        FEDORA-2015-3366 	duplicity 	2015-03-19
Fedora 	        FEDORA-2015-3497 	csync2 	        2015-03-19
Fedora 	        FEDORA-2015-3366 	csync2 	        2015-03-19
Gentoo 	        201605-04 	        rsync           2016-05-30

Example LWN vulnerability entry, from http://lwn.net/Vulnerabilities/637406/

Above is an example of one of their entries, showing which distributions issued advisories for which packages on which dates and including links to those advisories. The package listed at the top of the entry is usually the name of the first package whose advisory was added to the entry when it was first created. The CVEs list includes, if applicable, a list of the CVEs that were fixed in the initial advisory, and this is usually a list of vulnerabilities that are fixed together in one update. Any advisories that fix any of the issues in the list will be added to the entry. The description is usually taken from the first advisory, but sometimes comes from a bug tracker entry or an upstream advisory. The advisories themselves will often have links to bug tracker entries and/or references to other information about the vulnerabilities. When I create bug reports in our bug tracker, Bugzilla [10], I usually link directly to the advisory on the Linux distribution's own website, rather than LWN's copy. Most of these advisories are from Fedora [11], RedHat [12], Debian [13], Ubuntu [14], or OpenSuSE [15].

Upstream change logs and release announcements

When upstream developers release new software versions, they usually publish on their website a change log or release announcement of some sort, which vary in detail and readability and whether or not they highlight any security issues fixed. Sometimes this information isn't on their website, but is 6included with the code and can be found in a file named NEWS, CHANGES, or ChangeLog. It is important to read this information, especially for software that commonly fixes and/or announces security issues. The DistroWatch website lists recent updates to over 200 pieces of software, most of which we have packaged, on the left side of the front page of their site, including links to the upstream software's websites. Watching upstream release announcement mailing lists and following tools that track updated software packages, such as Mageia's own youri-check [16], can be helpful as well. Older Mageia update advisories [17] are a good source of links to upstream release announcement sources.

Reports from users

Sometimes, users may become aware of a security issue before I do, usually because they read about it somewhere, be that by reading the release announcements for a piece of software that they follow or by seeing it in a news article or on Twitter or some other third-party information source. In these cases, they may file a bug in our BugZilla themselves, e-mail me directly (either using my e-mail address or our security group alias, security@group.mageia.org), or say something about it in one of our IRC channels [18] on Freenode.

Determining its applicability to our products

Once I become aware of a security issue in some open-source software, next I must determine if it affects any of the packages in our distribution (including the development branch of the distribution and any currently supported stable versions). This includes determining if the affected code is present in any of the packages in our distribution. Determining that requires knowing the package names that may include the code, and the versions of the software in these packages that are affected by the issue. For most packages, the package name closely reflects the name of the software, and is the same or close to the same in most distributions. These determinations become more difficult when the code for one piece of software is bundled in the code for other packages and there can be multiple unrelated packages affected by the same issue, or when there are major mismatches between upstream software names and package names or between package names in different distributions. Examples of bundling issues include ffmpeg (a frequent source of security issues) being bundled in multiple other packages and the lzo and minilzo library code being bundled in several other packages [19]. Finding this bundled code can include relying on existing familiarity with the various software in the distribution, asking other 7packagers, and searching the packages (either the compiled ones or the source) for certain file names. Examples of package naming issues include two different pieces of software called libtorrent (whose package names are libtorrent and libtorrent-rasterbar) and a piece of software called Request Tracker, packaged in Debian as request-tracker but called rt in Mageia, which caused security updates to be missed for years in the latter case [20].

Once it has been determined that the affected software is present in the distribution, then it must be determined if the issue affects the version(s) of the software that we have. How this is determined depends on the information that is available. Sometimes, information about exactly what versions of the software are affected will be available, either in the CVE description, another distribution's bug or security issue tracker entry, or in the initial public announcement of the vulnerability. In these cases, it is usually easy to determine if any of the versions of the software that we have fall in the affected range. Other times, a link to the commit (the patch or source code change) that introduced the vulnerability will be available. In these cases, the result of the code changes made by that commit can be compared to the code that we have to see if it looks like our version comes after that commit was introduced into the code. As the commits also have time stamps, comparing the date of the commit to the date that the version we have was released or committed to our source code repository can sometimes be enough to determine whether or not our version of the code is affected. Similarly, for issues that have been or get fixed by the upstream developers, information about the version of the software in which it was fixed can be used to determine if our versions are new enough to already include the fix. In cases in which only the commit that fixed the issue is available, the code can be compared to determine if the fixed code is present in the copy that we have, or if the date indicates that our version is new enough. Lastly, in some cases a PoC (proof-of-concept, or data or program that can be used to reproduce the issue) is available, and our software can be directly tested to determine if it is vulnerable.

Finally, in some cases, the importance or severity of the security issue with respect to the use case of our products can be an important consideration. For severe and/or high-impact issues, it is important to know that priority must be given to fixing the issue. This must be conveyed to anyone else who may be involved in getting the update released, including packagers (who would be involved in packaging the update if I don't do it myself), the QA team (who test the updates), and the sysadmins who perform the final steps in releasing the update. In these cases, other distributions (such as RedHat or Ubuntu) will have rated the issue as having high or critical severity, and the issue may be attracting attention in the press, possibly including reports that the issue is presently being actively exploited. In 8other cases, the issue may be so minor that it is not worth addressing immediately. The fix may be committed to the repository such that it could be included in a future update. Alternatively, the issue may be so difficult to fix, or the risk of introducing regressions in attempting to fix the issue may be so great, that it is determined that it is not worth fixing the issue at all. Analysis by other distributions, security researchers, upstream developers, myself, and our own packagers, can help make these decisions.

Also, it is at this stage in the process where I file a bug report in our BugZilla, if one has not already been filed. Whether the bug is filed at the beginning or end of this stage depends on whether I need help from other packagers in determining whether our software is affected by the issue. If I can easily determine this myself, I will file the bug report once I have determined that we are affected. Otherwise, I will create the bug report first and include other packagers who are more familiar with the software and better equipped to determine the issue's impact on our distribution.

Locating a fix

Linux distributions have different approaches to how they prefer to fix bugs and security issues. Some distributions, such as RedHat and Debian, are more conservative and prefer to make the smallest change possible to address a specific issue, to minimize the risk of regressions, and will generally try to apply a patch to their existing version of the software to fix the issue. Other distributions, such as Fedora and Arch, are less conservative, and generally prefer to update to a newer version of the software which includes the fix. However, depending on the availability of fixed versions from the upstream developers, the difficulty of updating to newer versions, as well as the difficulty of patching older versions of the software, every distribution will use either of approaches in certain cases. Mageia tends to be conservative and generally prefers patching to updating when it is feasible [21].

Therefore, in most cases, fixing an issue requires locating a patch or patches to fix the issue(s). These can be found in various ways. If the upstream commit(s) to fix the issue is/are known, this is one source of patches. Updates released by other distributions or committed to their source code repositories are a frequent source of patches. In some cases, a third-party researcher will provide a proposed patch during the initial disclosure or discussion of the issue. This information would generally be available either on the oss-security mailing list or on a bug report in either a Linux distribution's or upstream's bug tracker. If newer versions of the software which fix the issue are available, this should be determined as well, as we may chose to update it instead of patching. This will often be the case in our development branch, but even for stable releases, we will chose this approach some times. Finally, for some vulnerabilities, they cannot actually be fixed (or at least cannot easily be fixed), but they can be mitigated by making a change to the software's default configuration, so we will collect this information if applicable as well.

Patches used by other distributions can generally be obtained either from their source packages or version control repositories. Source packages for Fedora, RedHat, and OpenSuSE are often available on their mirrors, and for Debian [22] and Ubuntu [23] , their package sources are available on their packages websites. Source code repositories are generally the best source, and these are available for Fedora [24], RedHat [25], Debian [26], and OpenSuSE [27] (as well as some others). Ubuntu [28] also has a security issue tracker which often has links to patches and other useful notes, and their advisories contains links to these tracker entries as well as their package sources.

When filing bug reports in our BugZilla for security issues, or shortly thereafter, I will usually include any of this information that I am aware of or can quickly determine. In some cases, I will rely on the packagers who are more familiar with the software to help find this information.

Implementing a fix and building the update

Once the possible methods of fixing the issue have been determined, then we must actually implement the fix. Sometimes this is trivial and sometimes it is the most time-consuming part of the process. This is where other packagers can be of the most help, and technically it is their responsibility to help with this for packages that they maintain. In practice, the amount of help I receive from different packagers varies quite a bit, and most of the work for implementing fixes for security issues is done by me, even though it is technically the responsibility [29] of the Packagers Team [30] and not the Security Team [31] (though I am a member of the Packagers Team, so I also contribute to this effort through that role).

As we generally prefer to patch security issues, we need a patch which will apply to the code that we have. This means that the section(s) of the code being changed by the patch match the code in the version of the software that we are patching. Patches from upstream that are for the same stable branch of the code that our version came from or patches from other Linux distributions for the same version of the software that we have tend to be the most helpful here. If a patch does not apply cleanly, often this can be fixed by what we call backporting or rediffing the patch, which means changing or regenerating 10the patch such that it applies to our version of the software. The first step is to try examining the sections (or hunks as they are called) of the patch which do not apply cleanly and comparing these sections of code to the code that we are patching. Often times, minor edits to the patch to make the code being changed exactly match what we have are enough to fix the patch such that it will apply cleanly and can be used. In some cases, this doesn't work or is impractical, so the approach is to manually make the same changes to the source code that the patch is intended to make and then regenerate the patch by using the diff command. With either of these approaches, understanding the actual code or code changes in the patch is not necessary as it is simply a matter of comparing and/or changing the text of the source code. There are some cases that are more complicated, where more substantial changes have been made to the style of the code between the version we have and the version the patch we have is for, but conceptually the code is still doing the same things and therefore the security issue still exists. In these cases, it is usually required that someone understand the code well enough to be able to recreate the changes that the patch is intending to make, to create a patch which performs the same changes and effectively fixes the issue.

In some cases, the patching approach is not ultimately chosen. This can be because patching is infeasible, too difficult or requires too much effort, or too risky because we cannot be sure that our adjusted or recreated patch effectively fixes the issue or will not cause regressions. It also can be because updating the software is generally preferable for that particular package. This can be because of the long-term maintenance cost and effort for this software or because the upstream developers have a known good history of providing updates which fix bugs without making other substantial changes or introducing regressions. These decisions are made on a case-by-case basis, and our past history in dealing with fixing issues in the affected software carries a lot of weight.

Finally, in some cases, an issue cannot or cannot easily be fixed, but can be mitigated by making changes to the default configuration for the software. Also, in some cases, such configuration changes are a required component of fixing the issue (possibly along with updating or patching the software). For example, the POODLE [32] (CVE-2014-3566) issue in the SSLv3 protocol was a structural flaw in the standard specification of the protocol, and therefore was literally not able to be fixed. Usage of this protocol had to be disabled in several packages' configurations, so that they would instead only use the newer TLS protocols. For the Drupal web application, there was at least one instance [33] where a configuration change had to be made in the package along with it being updated to a newer version to fix security issues.

Once the needed changes have been made to the affected packages to fix the issue(s), the changes are committed to our source code repository and the packages are submitted to our build system to build the fixed packages.

Writing an advisory for the update

Any proposed package update needs to include an advisory [34], which is the information which will be included in the update announcement that is intended for users and system administrators, describing the issues that have been fixed by the update as well as any other steps they may need to take besides just installing the updated packages. Advisories will usually (or always, in the case of security updates) include references, which are links to web pages which provide even more information about the issues. The advisory will first identify the name of the source package(s) for which security issues are being fixed. Then it will include descriptions of the security vulnerabilities, including CVE identifiers, if applicable. Information that users or system administrators may need to know about the update follows next. Finally, the references are listed. Also, the names and versions of the updated packages need to be listed in the bug report so that the QA Team knows what needs to be tested and the sysadmins will know which packages need to be pushed as updates once the testing has completed.

The descriptions of the security issues usually follow a format that is similar to that used on the CVE website [35], which identifies the affected software and version range, the type of security vulnerability, and the affected portion of the software's functionality or code. The type of vulnerability can include the type of software flaw (double-free, use-after-free, buffer over-read, buffer overflow, integer overflow, null pointer dereference, divide-by-zero, reachable assertion, etc) and the impact of the issue (denial of service, information disclosure, remote code execution, privilege escalation, etc). These descriptions can be taken from the CVE entries themselves, other vendors' advisories, other vendors' bug or security tracker entries, or upstream or third-party advisories or announcements for the issues. Sometimes we have to piece the available information together and write the descriptions ourselves. In cases where very little or no information about the vulnerabilities is available, the advisory is written in a more generic manner, usually just indicating that the package has been updated to a certain newer version and that it fixes bugs and/or security issues.

The references will include links to the CVE entries on the CVE website, other vendors' advisories, especially if they are the source of information that we used to write our descriptions or fix 12the packages or if they include useful additional information, upstream advisories, and third-party advisories if appropriate. If we updated the software to a newer version, we'll also include links to the release announcements, release notes, or change logs for the new versions if possible.

CVEs

CVEs are standard identifiers that are assigned to specific security vulnerabilities. They are used by software developers and system administrators to have a common name to use to refer to an issue, so that they can track the issue, collect information about it, and track the progress of it being fixed. CVEs are assigned by a CNA [36] (CVE Numbering Authority), which is primarily MITRE, but also includes software development organizations such as Mozilla, Google, RedHat, Debian, Oracle, and others. CVEs are usually assigned while an issue has not yet been publicly disclosed (i.e., it is embargoed as part of a coordinated disclosure process). However, for already-public issues in open-source software, CVEs are usually requested and assigned on the oss-security mailing list and assigned by MITRE. Distinct CVE identifiers are assigned for different vulnerability types, affected software versions, and for different code bases or implementations of certain functionality that may be affected [37].

Given the helpfulness of indexing security issues by CVEs for system administrators, other Linux distributions, the LWN vulnerabilities database, and others, we do our best to include relevant CVE identifiers in our advisories. Sometimes this can be a bit of a challenge. In some cases, since the CVE was assigned in private, it is not included in the information that made us aware of the issue, so we may not be aware of the CVE identifier at all, or may become aware of it very late in the process. For CVEs that are assigned in public on the oss-security list, there may be significant delays between when the issue was made public and the CVE was requested, or between when the CVE was requested and the assignment is made by MITRE. In some instances, the request never receives a response at all, and for a request that has not yet received a response, there is no way of knowing how long it may take for an assignment to be made or whether one will be made at all. This can cause delays for us in writing advisories and issuing updates. We have to use our best judgment in deciding how long to wait for a CVE assignment and when to move on with an advisory and update without a CVE identifier. When CVE identifiers become available after an update has been issued, the online version of the advisory on our wiki can (and ideally, should) be updated, but often is not as this is currently a cumbersome process. At the very least, when this happens, I note the CVE assignments in the bug report.

Testing (QA)

Once the updated packages have been built, the bug report is assigned to the QA Team (qa-bugs@ml.mageia.org) [38] so that the updated packages can be tested. Other than the advisory, any additional information that could help QA team members understand the issue, know how to reproduce it, or be able to test the software should be provided in the bug report. For reproducing the issue, sometimes a PoC (proof-of-concept) is available, which is an input data file, test program, or procedure that can be used to demonstrate the presence of a vulnerability. Usually, if one of these is available, we will have encountered it by this stage of the process, either in the initial announcement of the issue, an upstream bug report, or another Linux distribution's bug or security issue tracker. The QA team members may search for a PoC in various online databases [39] as well.

During the testing process, QA team members may ask for clarification on various things, such as how to use the affected software, how to use a PoC, whether their testing procedures are valid, whether their testing results are OK, or if they have identified that the updated packages either do not fix the issue they intend to fix or if they introduce regressions. In some cases, more packaging work may be required to create a viable update candidate. Members of the packaging and/or security teams should be involved and help with whatever is necessary throughout this stage of the process.

Every Linux distribution has a slightly different process for submitting update candidates and doing QA. Some do very little QA and do not treat it as a separate stage in the process, some do the QA offline or in private, and some do everything out in the open. Mageia primarily uses BugZilla to support this process, but also has the Mageia App Db [40] web application (which pulls all of its data from BugZilla) to give quick access to information needed about the current update candidates. Fedora has a web application called Bodhi [41] that is used during this process, which is integrated with, but separate from their BugZilla.

Releasing the update

Once the update candidate has been validated (i.e., approved) by the QA Team, a member of the Sysadmin Team [42] will see that the updated packages get pushed to the mirrors and that the advisory is posted to the website and e-mailed to the updates-announce mailing list [43] . Every Linux distribution has a similar mailing list for update announcements. Varying amounts of automation and manual work are involved in this last stage of the process for different Linux distributions.

Other considerations

As a member of the Packagers Team, I also participate in the development of the next version of the distribution. I help keep many of the core packages up to date, which can be helpful when security issues come about that have already been fixed in a released version of a software package. I also help keep other packagers informed about newer versions of software that may be available so that they can keep other packages up to date. I pay attention to packages that cause particular difficulty with regards to security updates, most often due to lack of maintainers (or lack of responsive maintainers), and work to have problematic packages removed from the distribution, or to bring more attention to them so that other packagers will help take responsibility for them. I track unfixed security issues that affect the development version, so that late in the release cycle we can work to try to get as many of these issues fixed as possible before releasing the next stable version of the distribution. I also track unfixed security issues [44] affecting our existing stable releases and keep the QA Team informed of upcoming updates (usually through weekly IRC meetings [45] or the qa-discuss mailing list [46]) and solicit help from other packagers in fixing security issues (usually through the mageia-dev mailing list [47]).

Conclusion

For Linux users or system administrators that are concerned about system security, it is important to understand the process through which security updates are created. There are many factors that can affect the timeliness of these updates being released. There is a lot of work involved, and most of the people doing the work are doing so on a volunteer basis, even if some of them are able to do some of the work as part of their day job. There is much that users of these systems can do to contribute [48] to the effort, and contributions can be made at almost every stage of the process, regardless of which Linux distribution that they are using.