The Red Hat SRT is looking for another member to investigate, triage, and respond to security vulnerabilities in Red Hat Enterprise Linux but also across other products and services. You'll join our diverse and enthusiastic team currently spread across eight different countries.

Sound interesting? See the full job description: Security Response Team Software Engineer. If you are interested please use the online application process.

Although the location is specified as the Czech Republic there is actually no specific restriction on the location of this position, and if you're right for the role you could be located at your nearest local world-wide Red Hat office, or possibly even remote.

When you look back, before they admitted to this practice, Microsoft actively used vulnerability counts in reports as a tool to discredit the security of open source distributions. Famously even Steve Ballmer participated in counting vulnerabilities using candy.

In other news, the Red Hat Enterprise Linux 4 risk report we release each year has been published (PDF). This whitepaper looks at the state of security for the first five years of Red Hat Enterprise Linux 4 from its release on February 15th, 2005. It includes metrics, key vulnerabilities, and the most common ways users were affected by security issues.

"Red Hat knew about 52% of the security vulnerabilities that we fixed in advance of them being publicly disclosed. The average time between Red Hat knowing about an issue and it being made public was 22 days (median 10 days).... A default installation of Red Hat Enterprise Linux 4 AS was vulnerable to 14 critical security issues over the entire five years. "

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.4, up to and including the 5.5 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you have installed or removed.

So for a default install, from release of 5.4 up to and including 5.5, we shipped 52 advisories to address 140 vulnerabilities. 5 advisories were rated critical, 14 were important, and the remaining 33 were moderate and low.

Or, for all packages, from release of 5.4 to and including 5.5, we shipped 75 advisories to address 187 vulnerabilities. 6 advisories were rated critical, 18 were important, and the remaining 51 were moderate and low.

Critical vulnerabilities

The 6 critical advisories were for 3 different packages. Given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting the memory flaws harder.

1. Four updates to Firefox (September 2009, October 2009, December 2009, February 2010) where a malicious web site could potentially run arbitrary code as the user running Firefox.
2. An update to kdelibs (November 2009), where a malicious web site could potentially run arbitrary code as the user running the Konqueror browser. kdelibs is not a default installation package.
3. An update to krb5, the Kerberos network authentication system (January 2010), where a remote KDC client could cause a crash or run arbitrary code as root. This issue only affected users that have configured and enabled krb5.

Updates to correct 24 out of the 25 critical vulnerabilities were available via Red Hat Network either the same day, or up to one calendar day after the issues were public. The update to fix Konqueror took us 4 calendar days.

Overall, for Red Hat Enterprise Linux 5 since release to date, 98% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Red Hat Enterprise Linux since 5.2 contained backported patches from the upstream Linux kernel to add the ability to restrict unprivileged mapping of low memory, designed to mitigate NULL pointer dereference flaws. In the last risk report we mentioned it was found that this protection was not sufficient, as a system with SELinux enabled was more permissive in allowing local users in the unconfined_t domain to map low memory areas even if the mmap_min_addr restriction is enabled. This is CVE-2009-2695 and was addressed in a kernel update in November 2009.

Previous updates

To compare these statistics with previous update releases we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the results illustrated by the following chart:

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 5.3 to 5.4, 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.

It all started when I bought her a Chumby for Christmas. A Chumby is a little bedside device that can act as an alarm clock as well as running flash-lite applets. What made it especially appealing is that you can write your own applets if you want, and the whole thing is Linux-based and designed to be hackable: they correctly abide by the GPL and have their sources available, you can build and install your own software, you can even enable ssh and have a remote shell if you want to. And with NTP the clock is always at the right time, since I really don't like having out-of-sync clocks around the house.

So it was time to connect another device to my wireless network: a device designed to be left on and permanently connected to the network, and having a connected microphone, in the bedroom. A quick look around the OS and I found that it had a web server accessible by default, and a pair of CGI scripts, written in shell script, running as root, that didn't correctly escape their input. (Hint: writing secure CGI scripts in shell is non-trivial).

With a bit of careful manipulation (to get around some character handling in the code) I had a remote root shell on a default Chumby and could stream audio from the microphone remotely. Oops. Not too big a deal though as it's unlikely you're going to have it directly connected to the internet, although with some social engineering, if you know someone with a Chumby, you could do a cunning cross-site scripting attack and get a reverse shell that way.

I contacted the Chumby folks and they dealt with this like an ideal vendor; acknowledging the issue, keeping in contact, and doing a security update. Good for them. I like this device and vendor so much I'm going to buy another Chumby, and a few colleagues from work are too.

But how many other devices do we connect to our networks without thinking about them, and how many folks outside of the security paranoid have properly secured and segmented wireless networks? I've got a IP wireless network CCTV camera and a VOIP phone system both which seem to be running Linux (and both which seem to have vulnerabilities) to worry about next although harder since both are closed systems which haven't released their source.

So for CVE database: CVE-2010-0418 is "Chumby One before 1.0.4 and Chumby Classic before 1.7.2 allows remote attackers to execute arbitrary commands via shell metacharacters in a carefully crafted request to the web interface". Reported 29 Dec 2009, vendor responded 29 Dec 2009, tested fix 3 Feb 2010, public and updates 4 Mar 2010.

The stories center around how open source software such as Firefox was able to produce updates to correct this issue just a few days after the Blackhat conference, while Microsoft still hasn't fixed it and are "investigating a possible vulnerability in Windows presented during Black Hat".

But the actual timeline is missing from these stories.

The NULL character certificate flaw (CVE-2009-2408) was actually disclosed by two researchers working independantly who both happened to present the work at the same conference, Blackhat, in July this year. Dan Kaminsky mentioned it as part of a series of PKI flaws he disclosed. Marlinspike had found the same flaw, but was able to demonstrate it in practice by managing to get a trusted Certificate Authority to sign such a malicious certificate.

The flaw was no Blackhat surprise; Dan Kaminsky actually found this issue many months ago and responsibly reported the issues to vendors including Red Hat, Microsoft, and Mozilla. We found out about this issue on 25th February 2009 and worked with Dan and some of the upstream projects on these issues in advance, so we had plenty of time to prepare updates and this is why we were able to have them ready to release just after the disclosure.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.3, up to and including the 5.4 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you have installed or removed.

So for a default install, from release of 5.3 up to and including 5.4, we shipped 51 advisories to address 166 vulnerabilities. 8 advisories were rated critical, 18 were important, and the remaining 25 were moderate and low.

Or, for all packages, from release of 5.3 to and including 5.4, we shipped 78 advisories to address 251 vulnerabilities. 9 advisories were rated critical, 28 were important, and the remaining 41 were moderate and low.

Critical vulnerabilities

The 9 critical advisories were for just 3 different packages. In all the cases below, given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.

1. Seven updates to Firefox (February, March 4th, March 27th, April 21st, April 27th, June, July ) where a malicious web site could potentially run arbitrary code as the user running Firefox.
2. An update to kdelibs (June), where a malicious web site could potentially run arbitrary code as the user running the Konqueror browser. kdelibs is not a default installation package.
3. An update to the NSS library (July), where a service could present a malicious SSL certificate causing a heap overflow which could potentially run arbitrary code as the user running a browser such as Firefox.

Updates to correct all of these critical vulnerabilities were available via Red Hat Network either the same day, or up to one calendar day after the issues were public.

In fact for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update available to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest during this period were several NULL pointer dereference kernel issues. NULL pointer dereference flaws in the Linux kernel can often be easily abused by a local unprivileged user to gain root privileges through the mapping of low memory pages and crafting them to contain valid malicious instructions:

• CVE-2009-2698 was public on August 24th and a working privilege escalation exploit was published about a week later. This issue was addressed for Red Hat Enterprise Linux 5 by a kernel update on August 24th.
• CVE-2009-2692 was public on August 13th and a working privilege escalation exploit was published the same day. This issue was addressed for Red Hat Enterprise Linux 5 by a kernel update on August 24th.
• CVE-2009-1897 was public on July 16th along with a working privilege escalation exploit. This issue affected only beta versions of the Red Hat Enterprise Linux 5.4 kernel and it was addressed prior to the release of Red Hat Enterprise Linux 5.4.

Red Hat Enterprise Linux since 5.2 has contained backported patches from the upstream Linux kernel to add the ability to restrict unprivileged mapping of low memory, designed to mitigate NULL pointer dereference flaws. However it was found that this protection was not sufficient, as a system with SELinux enabled is more permissive in allowing local users in the unconfined_t domain to map low memory areas even if the mmap_min_addr restriction is enabled. This is CVE-2009-2695 and will be addressed in a future kernel update.

Mitigations

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. From 5.3 to 5.4 there were three flaws blocked that would otherwise have required critical updates:

• CVE-2009-0692, a stack buffer overflow flaw in dhclient. FORTIFY_SOURCE protection detects the overflow and causes dhclient to exit with no security consequence. No security update for users of Red Hat Enterprise Linux 5 was needed.
• CVE-2009-1252 a buffer overflow flaw in NTP caught by FORTIFY_SOURCE. We issued an update as a remote attacker could still cause a denial of service.
• CVE-2009-0846, a uninitialized pointer free in krb5. glibc provides a hardened malloc/free implementation which mitigates the exploitability of this flaw, however we issued an update as a remote attacker could still cause a denial of service.

Previous updates

To compare these statistics with previous update releases we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the results illustrated by the following chart:

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.

MD2 signature verification

The first issue is that many web browsers still accept certificates with MD2 hash signatures, even though MD2 is no longer considered a cryptographically strong algorithm. This could make it easier for an attacker to create a malicious certificate that would be treated as trusted by a browser. It turns out that there are not many valid MD2 hash certificates around any more, and the main one that does exist is at the trusted root level anyway (and there is actually no need for a crypto library to verify the self-signature on a trusted root). So most vendors have chosen to address this issue by disabling MD2 completely for certificate verification. This is allocated CVE name CVE-2009-2409 ( single name for all affected products).

• OpenSSL. For upstream OpenSSL we have disabled MD2 support completely. This was done in two stages; the first was a patch in June 2009 that removed the redundant check of a trusted root self-signed certificate. Then in July, MD2 was totally disabled. So this issue does not affect OpenSSL 1.0.0 beta 3 or later. Although there have not yet been an upstream release of 0.9.8 containing this fix, a future OpenSSL 0.9.8 (after 0.9.8k) will disable MD2, probably in a few weeks.

• GnuTLS. The upstream GnuTLS library has for some time meant to have disabled MD2 support, although due to a broken patch it wasn't actually disabled correctly until January 2009. So this issue does not affect GnuTLS versions 2.6.4 and above, or GnuTLS versions 2.7.4 and above.

• NSS (and hence Firefox). The upstream NSS library since version 3.12.3 (April 2009) has disabled MD2 and MD4 by default (although legacy applications could turn it back on using an environment variable "NSS_ALLOW_WEAK_SIGNATURE_ALG" if they need to). Mozilla Firefox since version 3.5 has used this NSS version and therefore MD2 is disabled. I suspect this issue will get addressed in a future Firefox 3.0 update in the future too if they rebase to the new NSS.

There is no immediate panic to address this issue as a critical security issue, as in order for it to be exploited an attacker still has to create a MD2 collision with this root certificate; something that is as of today still a significant amount of effort.

My CVSS v2 base score for CVE-2009-2409 would be 2.6 (AV:N/AC:H/Au:N/C:N/I:P/A:N)

Differences in Common Name handling

This issue is about how Common Names are checked for validity by applications. For example if a server presents a certificate with two CN entries, how does the app validate those. Does it use the first one, the last one, or all of them?

• OpenSSL. OpenSSL provides an API that allow applications to check CN names any way they want. It turns out that, without sound guidance, applications have tended to do things differently. A summary of a few OpenSSL applications is in this Red Hat bugzilla comment. But as a CA should validate all CN names in a certificate being signing, these are really just bugs and do not have a security impact

Leading 0's in Common Name handling

The second issue is all about inconsistencies in the interpretation of subject x509 names in certificates. Specifically "issue 2b, subattack 1" is where a malicious certificate can contain leading 0's in the OID. The idea is that an attacker could add in some OID into a certificate that, when handled by the Certificate Authority, would appear to be some extension and ignored, but when handled by OpenSSL would appear to be the Common Name OID. So the attacker would present the certificate to a client application and it might think that the OID is actually a Common Name, and accept the certificate where it otherwise should not.

• OpenSSL. This is not a security issue for OpenSSL. Steve Henson explains: "OpenSSL does tolerate leading 0x80 but it does _not_ recognize this as commonName because the NID code checks for a precise match with the encoding. Attempts to print this out will never show commonName nor will attempts to look up using NID_commonName". However this will be addressed as a bug fix in the future.

• NSS (and hence Firefox). NSS is noted in the paper as having a similar issue, but again it's not fooled into treating the OID as a Common Name so this is not a security issue (and therefore I didn't check if this is already fixed in the new upstream NSS).

OID overflow in Common Name handling

"issue 2b, subattack 2" is where a malicious certificate can have a very large integer in the OID. The idea is that an attacker could add in some OID into a certificate that, when handled by the CA, would appear to be some extension and ignored, but when handled by OpenSSL would overflow and appear to be the Common Name OID. So the attacker would present the certificate to a client application using OpenSSL and it might think that the OID is actually a Common Name, and accept the certificate where it otherwise should not.

• OpenSSL. This issue was actually fixed upstream in September 2006 in OpenSSL 0.9.8d by switching to using the bignum library for handling the OID. Even for older versions though it's really not a security issue for the same reason as given earlier: the OpenSSL NID code checks for a precise match with the encoding. So attempts to print this out will never show it being a Common Name, nor will attempts to look it up as a Common Name succeed.

NULL bytes in Common Name handling

"issue 2, attack 2c" is regarding NULL terminators in a Common Name field. If an attacker is able to get a carefully-crafted certificate signed by a Certificate Authority trusted by a browser, the attacker could use the certificate during a man-in-the-middle attack and potentially confuse the browser into accepting it by mistake.

• NSS (and hence Firefox). This issue affected NSS and is assigned CVE-2009-2408. The upstream NSS library since version 3.12.3 (April 2009) has fixes to address this issue. Therefore Firefox 3.5 is not affected.

My CVSS v2 base score for CVE-2009-2408 would be 4.3 (AV:N/AC:M/Au:N/C:N/I:P/A:N)

OpenSSL 'compat mode' subject name injection

"issue 2d" is how the OpenSSL command line utility will output unescaped subject X509 lines to standard output. So if some utility runs the openssl application from the command line and parses the text output, and if an attacker can craft a malicious certificate in such a way they fool a CA into signing it, they could present it to the utility and possibly fool that utility into thinking fields were different to what they actually are, perhaps allowing the certificate to be accepted as legitimate.

• OpenSSL. This attack requires that some utility will parse the output of OpenSSL command line using the default 'compat' mode. Applications should never do this. Upstream OpenSSL are unlikely to address this issue directly, although in the future the default output mode perhaps could be changed to something other than 'compat', and it's likely a documentation update will remind users that parsing the output of running such an openssl command is not the right way to use OpenSSL.

OpenSSL ASN1 printing crash

• OpenSSL. This was reported by Dan to OpenSSL and fixed in March 2009. This was allocated CVE name CVE-2009-0590. Therefore OpenSSL 0.9.8k and later contains a fix for this issue.

My CVSS v2 base score for CVE-2009-0590 would be 2.6 (AV:N/AC:H/Au:N/C:N/I:N/A:P)

From time to time I publish metrics on vulnerabilities that affect Red Hat Enterprise Linux. One of the more interesting metrics looks at how far in advance we know about the vulnerabilities we fix, and from where we get that information. This post is abstracted from the upcoming "4 years of Enterprise Linux 4" risk report

For every fixed vulnerability across every package and every severity in Enterprise Linux 4 AS in the first 4 years of its life, we determined if the flaw was something we knew about a day or more in advance of it being publicly disclosed, and how we found out about the flaw.

For vulnerabilities which are already public when we first hear about them we still track the source as it's a useful internal indicator on where the security response team should focus their efforts.

So from this data, Red Hat knew about 51% of the security vulnerabilities that we fixed at least a day in advance of them being publicly disclosed. For those issues, the average notice was 21 calendar days, although the median was much lower, with half the private issues having advance notice of 9 days or less.

The chart below shows the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.2, up to and including the 5.3 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). So, for a given installation, the number of packages and vulnerabilities will probably be somewhere between the two.

So for a default install, from release of 5.2 up to and including 5.3, we shipped 45 advisories to address 127 vulnerabilities. 7 advisories were rated critical, 21 were important, and the remaining 17 were moderate and low.

For all packages, from release of 5.2 to and including 5.3, we shipped 61 advisories to address 181 vulnerabilities. 7 advisories were rated critical, 28 were important, and the remaining 26 were moderate and low.

The 7 critical advisories were for just 3 different packages:

1. Five updates to Firefox (July, July, September, November, December) where a malicious web site could potentially run arbitrary code as the user running Firefox. Given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.
2. An update to Samba (May), where a remote attacker who can connect and send a print request to a Samba server could cause a heap overflow. The Red Hat Security Response Team believes it would be hard to remotely exploit this issue to execute arbitrary code due to the default enabled SELinux targeted policy and the default enabled SELinux memory protection tests. We are not aware of any public exploit for this issue.
3. An update to OpenSSH (August), provided to mitigate an intrusion into certain Red Hat computer systems. The attacker was able to sign a small number of tampered packages but they were not distributed on the Red Hat Network. We classified this update as critical to ensure any tampered packages would be replaced with official packages.

Although not of critical severity, also of interest during this period were the spoofing attacks on DNS servers. We provided an update to BIND (July) adding source port randomization to help mitigate these attacks.

Updates to correct all of these critical vulnerabilities (as well as migitate the BIND issue) were available via Red Hat Network either the same day, or one calendar day after the issues were public.

In fact for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update available to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

To compare this with the last updates we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. For 5.2 to 5.3 there were two flaws blocked that would otherwise have required updates:

1. A double-free flaw in unzip. The glibc pointer checking limited the exploitability of this issue to just a crash of unzip, a client application, which does not have security implications. No security update was needed.
2. Two format string flaws in c++filt. The format string protection caused these issues to have no security implications. No security update was needed.

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

Actually this isn't surprising and is exactly what I'd expect; it's all down to third party applications.

Let's say you're browsing the web. It's more than likely that at some point you'll want to view some PDF files, watch some Flash content, or play a Java game. Those tasks are all dealt with by third party applications, although to the end user it's all part of the browser experience. Since your system is only as secure as its weakest link, you need to manage security updates for those third party applications just as carefully as you manage security updates for the rest of your system. That's why Adobe Reader, Java, Flash, and all the myriad of other applications you've installed in order to make your system useful have their own update mechanisms. Some applications on Windows will 'phone home' when they are run and check to see if they need to be updated, others deploy services that sit in the background looking for updates from time to time, others even check every time your system starts. Many don't get automated updates at all.

How do you deal with all that risk? I believe it's possible by providing an OS distribution which includes all the bits you'll likely need to make a useful computing environment, thereby taking away that update uncertainty. Red Hat ship several PDF viewers in our distributions for example, but we also ship (in an Extras channel) Adobe Reader. Our Security Response Team are monitoring for security issues in everything we ship, all the third party applications, and providing a single point of contact, a single notification system, and a single way to get the updates.

If Microsoft knew that say 25% of all their users installed Firefox, wouldn't they be better bundling it and providing their centralised automated updates for it, to reduce their customers overall risk? They do already bundle some third party applications, although it's been with mixed success as we found 3 years ago when they didn't provide security fixes for bundled Flash (ZDNet coverage).

This is, in part, why you've not seen me respond recently to the Vista security reports which compare vulnerability counts. In these reports they use a cut-down minimal Red Hat Enterprise Linux installation in order to make it look more like Windows for the comparisons. But this is completely backwards -- the fact that we're including and fixing the flaws using a common process in so much third party software is actually helping reduce the risk and protect real customers. For example we could easily cut our vulnerability count by shipping only one PDF viewer instead of four. But if we know that these other viewers are going to get installed by the customer anyway all we've done is to hide the vulnerability count elsewhere, and you've made the customers overall risk increase.

So it may seem counter-intuitive but we should ship as much third party applications (that we know people use) as we can, because a single managed security update and notification process will decrease a users overall risk. The fewer third party applications that users have to get from elsewhere and install and manage for themselves the better in my opinion.

Hi! I'm Mark Cox. This blog gives my thoughts and opinions on my security work, open source, fedora, home automation, and other topics.

pics from my twitter:

popular tags: [all], apache, apachecon, apacheweek, bryce, cve, fedora, financial, geocaching, gps, ha, metrics, microsoft, nashville, north carolina, red hat summit, security, trips