When you look back, before they admitted to this practice, Microsoft actively used vulnerability counts in reports as a tool to discredit the security of open source distributions. Famously even Steve Ballmer participated in counting vulnerabilities using candy.

In other news, the Red Hat Enterprise Linux 4 risk report we release each year has been published (PDF). This whitepaper looks at the state of security for the first five years of Red Hat Enterprise Linux 4 from its release on February 15th, 2005. It includes metrics, key vulnerabilities, and the most common ways users were affected by security issues.

"Red Hat knew about 52% of the security vulnerabilities that we fixed in advance of them being publicly disclosed. The average time between Red Hat knowing about an issue and it being made public was 22 days (median 10 days).... A default installation of Red Hat Enterprise Linux 4 AS was vulnerable to 14 critical security issues over the entire five years. "

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.4, up to and including the 5.5 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you have installed or removed.

So for a default install, from release of 5.4 up to and including 5.5, we shipped 52 advisories to address 140 vulnerabilities. 5 advisories were rated critical, 14 were important, and the remaining 33 were moderate and low.

Or, for all packages, from release of 5.4 to and including 5.5, we shipped 75 advisories to address 187 vulnerabilities. 6 advisories were rated critical, 18 were important, and the remaining 51 were moderate and low.

Critical vulnerabilities

The 6 critical advisories were for 3 different packages. Given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting the memory flaws harder.

1. Four updates to Firefox (September 2009, October 2009, December 2009, February 2010) where a malicious web site could potentially run arbitrary code as the user running Firefox.
2. An update to kdelibs (November 2009), where a malicious web site could potentially run arbitrary code as the user running the Konqueror browser. kdelibs is not a default installation package.
3. An update to krb5, the Kerberos network authentication system (January 2010), where a remote KDC client could cause a crash or run arbitrary code as root. This issue only affected users that have configured and enabled krb5.

Updates to correct 24 out of the 25 critical vulnerabilities were available via Red Hat Network either the same day, or up to one calendar day after the issues were public. The update to fix Konqueror took us 4 calendar days.

Overall, for Red Hat Enterprise Linux 5 since release to date, 98% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Red Hat Enterprise Linux since 5.2 contained backported patches from the upstream Linux kernel to add the ability to restrict unprivileged mapping of low memory, designed to mitigate NULL pointer dereference flaws. In the last risk report we mentioned it was found that this protection was not sufficient, as a system with SELinux enabled was more permissive in allowing local users in the unconfined_t domain to map low memory areas even if the mmap_min_addr restriction is enabled. This is CVE-2009-2695 and was addressed in a kernel update in November 2009.

Previous updates

To compare these statistics with previous update releases we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the results illustrated by the following chart:

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 5.3 to 5.4, 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.

During the creation and review of the list we spent some time to see how closely last years list matched the types of flaws we deal with at Red Hat. We first looked at all the issues that Red Hat fixed across our entire product portfolio in the 2009 calendar year and filtered out those that had the highest severity. All our 2009 vulnerabilities have CVSS scores, so we filtered on those that have a CVSS base score of 7.0 or above^[1].

There were 22 vulnerabilities that matched, and we mapped each one to the most appropriate CWE. This gives us 11 flaw types which led to the most severe flaws affecting Red Hat in 2009:

CWE	CWE Description	CWE/SANS top 25?	Number of Vulnerabilities
CWE-476	NULL Pointer Dereference	No (on cusp)	6
CWE-120	Buffer Copy without Checking Size of Input	Yes	3
CWE-129	Improper Validation of Array Index	Yes	3
CWE-131	Incorrect Calculation of Buffer Size	Yes	3
CWE-78	OS Command Injection	Yes	1
CWE-285	Improper Access Control (Authorization)	Yes	1
CWE-362	Race Condition	Yes	1
CWE-330	Use of Insufficiently Random Values	No (on cusp)	1
CWE-590	Free of Memory not on the Heap	No	1
CWE-672	Use of a Resource after Expiration or Release	No (on cusp)	1
CWE-772	Missing Release of Resource after Effective Lifetime	No (on cusp)	1

10 of the 11 CWE are mentioned in the 2010 CWE/SANS document, although 4 of them are on "the cusp" and didn't make it into the top 25.

This quick review shows us that 2009 was the year of the kernel NULL pointer dereference flaw, as they could allow local untrusted users to gain privileges, and several public exploits to do just that were released. For Red Hat, interactions with SELinux prevented them being able to be easily mitigated, until the end of the year when we provided updates. Now, in 2010, the upstream Linux kernel and many vendors ship with protections to prevent kernel NULL pointers leading to privilege escalation. So although 2009 was the year where CWE-476 mattered to Linux administrators, it didn't make the SANS/CWE top 25 as this flaw type should not lead to severe issues (as long as the protections remain sufficient).

Here is a breakdown with the complete data set to show the CVSS scores and packages affected:

CVE	CWE	top 25?	CVSS base	Fixed in
CVE-2008-5182	CWE-362	Yes	7.2	Red Hat Enterprise Linux 5 (kernel)
CVE-2009-0065	CWE-129	Yes	8.3	Red Hat Enterprise Linux 4,5,MRG (kernel)
CVE-2009-0692	CWE-120	Yes	8.3	Red Hat Enterprise Linux 3,4 (dhcp)
CVE-2009-0778	CWE-772	No (on cusp)	7.1	Red Hat Enterprise Linux 5 (kernel)
CVE-2009-0846	CWE-590	No	9.3	Red Hat Enterprise Linux 2.1, 3 (krb5) [2]
CVE-2009-1185	CWE-131	Yes	7.2	Red Hat Enterprise Linux 5 (udev)
CVE-2009-1385	CWE-129	Yes	7.1	Red Hat Enterprise Linux 3,4,5,MRG (kernel)
CVE-2009-1439	CWE-131	Yes	7.1	Red Hat Enterprise Linux 4,5,MRG (kernel)
CVE-2009-1579	CWE-78	Yes	7.5	Red Hat Enterprise Linux 3,4,5 (squirrelmail)
CVE-2009-1633	CWE-131	Yes	7.1	Red Hat Enterprise Linux 4,5,MRG (kernel)
CVE-2009-2406	CWE-120	Yes	7.2	Red Hat Enterprise Linux 5 (kernel)
CVE-2009-2407	CWE-120	Yes	7.2	Red Hat Enterprise Linux 5 (kernel)
CVE-2009-2692	CWE-476	No (on cusp)	7.2	Red Hat Enterprise Linux 3,4,5,MRG (kernel)
CVE-2009-2694	CWE-129	Yes	7.5	Red Hat Enterprise Linux 3,4,5 (pidgin)
CVE-2009-2698	CWE-476	No (on cusp)	7.2	Red Hat Enterprise Linux 3,4,5 (kernel)
CVE-2009-2848	CWE-672	No (on cusp)	7.2	Red Hat Enterprise Linux 3,4,5,MRG (kernel)
CVE-2009-2908	CWE-476	No (on cusp)	7.2	Red Hat Enterprise Linux 5 (kernel)
CVE-2009-3238	CWE-330	No (on cusp)	7.8	Red Hat Enterprise Linux 4,5,MRG (kernel)
CVE-2009-3290	CWE-285	Yes	7.2	Red Hat Enterprise Linux 5 (kvm)
CVE-2009-3547	CWE-476	No (on cusp)	7.2	Red Hat Enterprise Linux 3,4,5,MRG (kernel)
CVE-2009-3620	CWE-476	No (on cusp)	7.2	Red Hat Enterprise Linux 4,5,MRG (kernel)
CVE-2009-3726	CWE-476	No (on cusp)	7.2	Red Hat Enterprise Linux 5,MRG (kernel)

[1] NIST NVD rate vulnerabilities as "High" severity if they have a CVSS base score of 7.0-10.0. This ends up excluding flaws in web browsers such as Firefox which can have a maximum CVSS base score of 6.8.

[2] Red Hat Enterprise Linux 4 and 5 were also affected by this vulnerability, but with a lower CVSS base score of 4.3, due to the extra runtime pointer checking.

The stories center around how open source software such as Firefox was able to produce updates to correct this issue just a few days after the Blackhat conference, while Microsoft still hasn't fixed it and are "investigating a possible vulnerability in Windows presented during Black Hat".

But the actual timeline is missing from these stories.

The NULL character certificate flaw (CVE-2009-2408) was actually disclosed by two researchers working independantly who both happened to present the work at the same conference, Blackhat, in July this year. Dan Kaminsky mentioned it as part of a series of PKI flaws he disclosed. Marlinspike had found the same flaw, but was able to demonstrate it in practice by managing to get a trusted Certificate Authority to sign such a malicious certificate.

The flaw was no Blackhat surprise; Dan Kaminsky actually found this issue many months ago and responsibly reported the issues to vendors including Red Hat, Microsoft, and Mozilla. We found out about this issue on 25th February 2009 and worked with Dan and some of the upstream projects on these issues in advance, so we had plenty of time to prepare updates and this is why we were able to have them ready to release just after the disclosure.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.3, up to and including the 5.4 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you have installed or removed.

So for a default install, from release of 5.3 up to and including 5.4, we shipped 51 advisories to address 166 vulnerabilities. 8 advisories were rated critical, 18 were important, and the remaining 25 were moderate and low.

Or, for all packages, from release of 5.3 to and including 5.4, we shipped 78 advisories to address 251 vulnerabilities. 9 advisories were rated critical, 28 were important, and the remaining 41 were moderate and low.

Critical vulnerabilities

The 9 critical advisories were for just 3 different packages. In all the cases below, given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.

1. Seven updates to Firefox (February, March 4th, March 27th, April 21st, April 27th, June, July ) where a malicious web site could potentially run arbitrary code as the user running Firefox.
2. An update to kdelibs (June), where a malicious web site could potentially run arbitrary code as the user running the Konqueror browser. kdelibs is not a default installation package.
3. An update to the NSS library (July), where a service could present a malicious SSL certificate causing a heap overflow which could potentially run arbitrary code as the user running a browser such as Firefox.

Updates to correct all of these critical vulnerabilities were available via Red Hat Network either the same day, or up to one calendar day after the issues were public.

In fact for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update available to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest during this period were several NULL pointer dereference kernel issues. NULL pointer dereference flaws in the Linux kernel can often be easily abused by a local unprivileged user to gain root privileges through the mapping of low memory pages and crafting them to contain valid malicious instructions:

• CVE-2009-2698 was public on August 24th and a working privilege escalation exploit was published about a week later. This issue was addressed for Red Hat Enterprise Linux 5 by a kernel update on August 24th.
• CVE-2009-2692 was public on August 13th and a working privilege escalation exploit was published the same day. This issue was addressed for Red Hat Enterprise Linux 5 by a kernel update on August 24th.
• CVE-2009-1897 was public on July 16th along with a working privilege escalation exploit. This issue affected only beta versions of the Red Hat Enterprise Linux 5.4 kernel and it was addressed prior to the release of Red Hat Enterprise Linux 5.4.

Red Hat Enterprise Linux since 5.2 has contained backported patches from the upstream Linux kernel to add the ability to restrict unprivileged mapping of low memory, designed to mitigate NULL pointer dereference flaws. However it was found that this protection was not sufficient, as a system with SELinux enabled is more permissive in allowing local users in the unconfined_t domain to map low memory areas even if the mmap_min_addr restriction is enabled. This is CVE-2009-2695 and will be addressed in a future kernel update.

Mitigations

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. From 5.3 to 5.4 there were three flaws blocked that would otherwise have required critical updates:

• CVE-2009-0692, a stack buffer overflow flaw in dhclient. FORTIFY_SOURCE protection detects the overflow and causes dhclient to exit with no security consequence. No security update for users of Red Hat Enterprise Linux 5 was needed.
• CVE-2009-1252 a buffer overflow flaw in NTP caught by FORTIFY_SOURCE. We issued an update as a remote attacker could still cause a denial of service.
• CVE-2009-0846, a uninitialized pointer free in krb5. glibc provides a hardened malloc/free implementation which mitigates the exploitability of this flaw, however we issued an update as a remote attacker could still cause a denial of service.

Previous updates

To compare these statistics with previous update releases we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the results illustrated by the following chart:

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.

From time to time I publish metrics on vulnerabilities that affect Red Hat Enterprise Linux. One of the more interesting metrics looks at how far in advance we know about the vulnerabilities we fix, and from where we get that information. This post is abstracted from the upcoming "4 years of Enterprise Linux 4" risk report

For every fixed vulnerability across every package and every severity in Enterprise Linux 4 AS in the first 4 years of its life, we determined if the flaw was something we knew about a day or more in advance of it being publicly disclosed, and how we found out about the flaw.

For vulnerabilities which are already public when we first hear about them we still track the source as it's a useful internal indicator on where the security response team should focus their efforts.

So from this data, Red Hat knew about 51% of the security vulnerabilities that we fixed at least a day in advance of them being publicly disclosed. For those issues, the average notice was 21 calendar days, although the median was much lower, with half the private issues having advance notice of 9 days or less.

The chart below shows the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.2, up to and including the 5.3 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). So, for a given installation, the number of packages and vulnerabilities will probably be somewhere between the two.

So for a default install, from release of 5.2 up to and including 5.3, we shipped 45 advisories to address 127 vulnerabilities. 7 advisories were rated critical, 21 were important, and the remaining 17 were moderate and low.

For all packages, from release of 5.2 to and including 5.3, we shipped 61 advisories to address 181 vulnerabilities. 7 advisories were rated critical, 28 were important, and the remaining 26 were moderate and low.

The 7 critical advisories were for just 3 different packages:

1. Five updates to Firefox (July, July, September, November, December) where a malicious web site could potentially run arbitrary code as the user running Firefox. Given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.
2. An update to Samba (May), where a remote attacker who can connect and send a print request to a Samba server could cause a heap overflow. The Red Hat Security Response Team believes it would be hard to remotely exploit this issue to execute arbitrary code due to the default enabled SELinux targeted policy and the default enabled SELinux memory protection tests. We are not aware of any public exploit for this issue.
3. An update to OpenSSH (August), provided to mitigate an intrusion into certain Red Hat computer systems. The attacker was able to sign a small number of tampered packages but they were not distributed on the Red Hat Network. We classified this update as critical to ensure any tampered packages would be replaced with official packages.

Although not of critical severity, also of interest during this period were the spoofing attacks on DNS servers. We provided an update to BIND (July) adding source port randomization to help mitigate these attacks.

Updates to correct all of these critical vulnerabilities (as well as migitate the BIND issue) were available via Red Hat Network either the same day, or one calendar day after the issues were public.

In fact for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update available to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

To compare this with the last updates we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. For 5.2 to 5.3 there were two flaws blocked that would otherwise have required updates:

1. A double-free flaw in unzip. The glibc pointer checking limited the exploitability of this issue to just a crash of unzip, a client application, which does not have security implications. No security update was needed.
2. Two format string flaws in c++filt. The format string protection caused these issues to have no security implications. No security update was needed.

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

Actually this isn't surprising and is exactly what I'd expect; it's all down to third party applications.

Let's say you're browsing the web. It's more than likely that at some point you'll want to view some PDF files, watch some Flash content, or play a Java game. Those tasks are all dealt with by third party applications, although to the end user it's all part of the browser experience. Since your system is only as secure as its weakest link, you need to manage security updates for those third party applications just as carefully as you manage security updates for the rest of your system. That's why Adobe Reader, Java, Flash, and all the myriad of other applications you've installed in order to make your system useful have their own update mechanisms. Some applications on Windows will 'phone home' when they are run and check to see if they need to be updated, others deploy services that sit in the background looking for updates from time to time, others even check every time your system starts. Many don't get automated updates at all.

How do you deal with all that risk? I believe it's possible by providing an OS distribution which includes all the bits you'll likely need to make a useful computing environment, thereby taking away that update uncertainty. Red Hat ship several PDF viewers in our distributions for example, but we also ship (in an Extras channel) Adobe Reader. Our Security Response Team are monitoring for security issues in everything we ship, all the third party applications, and providing a single point of contact, a single notification system, and a single way to get the updates.

If Microsoft knew that say 25% of all their users installed Firefox, wouldn't they be better bundling it and providing their centralised automated updates for it, to reduce their customers overall risk? They do already bundle some third party applications, although it's been with mixed success as we found 3 years ago when they didn't provide security fixes for bundled Flash (ZDNet coverage).

This is, in part, why you've not seen me respond recently to the Vista security reports which compare vulnerability counts. In these reports they use a cut-down minimal Red Hat Enterprise Linux installation in order to make it look more like Windows for the comparisons. But this is completely backwards -- the fact that we're including and fixing the flaws using a common process in so much third party software is actually helping reduce the risk and protect real customers. For example we could easily cut our vulnerability count by shipping only one PDF viewer instead of four. But if we know that these other viewers are going to get installed by the customer anyway all we've done is to hide the vulnerability count elsewhere, and you've made the customers overall risk increase.

So it may seem counter-intuitive but we should ship as much third party applications (that we know people use) as we can, because a single managed security update and notification process will decrease a users overall risk. The fewer third party applications that users have to get from elsewhere and install and manage for themselves the better in my opinion.

The graph below shows the total number of security updates issued for Red Hat Enterprise Linux 5 Server starting at 5.1 up to and including the 5.2 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). So, for a given installation, the number of packages and vulnerabilities will probably be somewhere between the two.

So for a default install, from release of 5.1 up to and including 5.2, we shipped 46 updates to address 119 vulnerabilities. 8 advisories were rated critical, 24 were important, and the remaining 14 were moderate and low.

For all packages, from release of 5.1 to and including 5.2, we shipped 62 updates to address 179 vulnerabilities. 9 advisories were rated critical, 29 were important, and the remaining 24 were moderate and low.

The nine critical updates were in five different packages:

1. Four updates to Firefox (November, February, March, April) where a malicious web site could potentially run arbitrary code as the user running Firefox. Given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.
2. An update to the GnuTLS library (May), where a remote attacker who can connect to a server making use of GnuTLS could cause a buffer overflow. In Red Hat Enterprise Linux 5, the CUPS print server uses GnuTLS.

3. An update to MIT Kerberos (March), where a remote attacker who can conect to the krb5kdc or kadmind services could cause a buffer overflow.

4. An update to OpenPegasus (January), where a remote attacker who can connect to OpenPegasus could cause a buffer overflow. The Red Hat Security Response Team believes that it would be hard to remotely exploit this issue to execute arbitrary code, due to the default SELinux targeted policy, and the default SELinux memory protection tests.

5. Two updates to Samba (November, December) where a remote attacker who can connect to the Samba port could cause buffer overflows. In addition to ExecShield making this harder to exploit, the impact of any sucessful exploit would be reduced as Samba is constrained by an SELinux targeted policy (enabled by default).

Updates to correct all of these critical issues were available via Red Hat Network either the same day, or one calendar day after the issues were public.

To get a better idea of risk we need to look not only at the vulnerabilities but also the exploits written for those vulnerabilities. A proof of concept exploit exists publicly for one of the Samba flaws, CVE-2007-6015, but we are not aware of public exploits for any other of those critical vulnerabilities. Also of high risk was an important "zero-day" exploit affecting the Linux kernel where a local unprivileged user could gain root privileges. Red Hat Enterprise Linux 5.1 was affected and a fix was available two calendar days after public disclosure.

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. For the period of this study there were two flaws blocked that would otherwise have required updates:

1. A double-free flaw in CUPS. The glibc pointer checking limited the exploitability of this issue to just a crash of CUPS and not the ability to execute arbitrary code. code execution. We still issued an update, as a remote attacker could trigger this flaw and cause CUPS to crash.
2. An uninitialized pointer free flaw in unzip, caught by the glibc pointer checking. As exploitation of this flaw results in just a crash of a user application, no updates were needed.

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions or distributions -- for example, a default install of Red Hat Enterprise 4AS did not include Firefox. You can get the results I presented above for yourself by using our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severities.

My full risk report was published yesterday in Red Hat Magazine and reveals the state of security since the release of Red Hat Enterprise Linux 4 including metrics, key vulnerabilities, and the most common ways users were affected by security issues.

"Red Hat knew about 49% of the security vulnerabilities that we fixed in advance of them being publicly disclosed. For those issues, the average notice was 21 calendar days, although the median was much lower, with half the private issues having advance notice of 8 days or less."

Hi! I'm Mark Cox. This blog gives my thoughts and opinions on my security work, open source, fedora, home automation, and other topics.

pics from my twitter:

popular tags: [all], apache, apachecon, apacheweek, bryce, cve, fedora, financial, geocaching, gps, ha, metrics, microsoft, nashville, north carolina, red hat summit, security, trips