The stories center around how open source software such as Firefox was able to produce updates to correct this issue just a few days after the Blackhat conference, while Microsoft still hasn't fixed it and are "investigating a possible vulnerability in Windows presented during Black Hat".

But the actual timeline is missing from these stories.

The NULL character certificate flaw (CVE-2009-2408) was actually disclosed by two researchers working independantly who both happened to present the work at the same conference, Blackhat, in July this year. Dan Kaminsky mentioned it as part of a series of PKI flaws he disclosed. Marlinspike had found the same flaw, but was able to demonstrate it in practice by managing to get a trusted Certificate Authority to sign such a malicious certificate.

The flaw was no Blackhat surprise; Dan Kaminsky actually found this issue many months ago and responsibly reported the issues to vendors including Red Hat, Microsoft, and Mozilla. We found out about this issue on 25th February 2009 and worked with Dan and some of the upstream projects on these issues in advance, so we had plenty of time to prepare updates and this is why we were able to have them ready to release just after the disclosure.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.3, up to and including the 5.4 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you have installed or removed.

So for a default install, from release of 5.3 up to and including 5.4, we shipped 51 advisories to address 166 vulnerabilities. 8 advisories were rated critical, 18 were important, and the remaining 25 were moderate and low.

Or, for all packages, from release of 5.3 to and including 5.4, we shipped 78 advisories to address 251 vulnerabilities. 9 advisories were rated critical, 28 were important, and the remaining 41 were moderate and low.

Critical vulnerabilities

The 9 critical advisories were for just 3 different packages. In all the cases below, given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.

1. Seven updates to Firefox (February, March 4th, March 27th, April 21st, April 27th, June, July ) where a malicious web site could potentially run arbitrary code as the user running Firefox.
2. An update to kdelibs (June), where a malicious web site could potentially run arbitrary code as the user running the Konqueror browser. kdelibs is not a default installation package.
3. An update to the NSS library (July), where a service could present a malicious SSL certificate causing a heap overflow which could potentially run arbitrary code as the user running a browser such as Firefox.

Updates to correct all of these critical vulnerabilities were available via Red Hat Network either the same day, or up to one calendar day after the issues were public.

In fact for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update available to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest during this period were several NULL pointer dereference kernel issues. NULL pointer dereference flaws in the Linux kernel can often be easily abused by a local unprivileged user to gain root privileges through the mapping of low memory pages and crafting them to contain valid malicious instructions:

• CVE-2009-2698 was public on August 24th and a working privilege escalation exploit was published about a week later. This issue was addressed for Red Hat Enterprise Linux 5 by a kernel update on August 24th.
• CVE-2009-2692 was public on August 13th and a working privilege escalation exploit was published the same day. This issue was addressed for Red Hat Enterprise Linux 5 by a kernel update on August 24th.
• CVE-2009-1897 was public on July 16th along with a working privilege escalation exploit. This issue affected only beta versions of the Red Hat Enterprise Linux 5.4 kernel and it was addressed prior to the release of Red Hat Enterprise Linux 5.4.

Red Hat Enterprise Linux since 5.2 has contained backported patches from the upstream Linux kernel to add the ability to restrict unprivileged mapping of low memory, designed to mitigate NULL pointer dereference flaws. However it was found that this protection was not sufficient, as a system with SELinux enabled is more permissive in allowing local users in the unconfined_t domain to map low memory areas even if the mmap_min_addr restriction is enabled. This is CVE-2009-2695 and will be addressed in a future kernel update.

Mitigations

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. From 5.3 to 5.4 there were three flaws blocked that would otherwise have required critical updates:

• CVE-2009-0692, a stack buffer overflow flaw in dhclient. FORTIFY_SOURCE protection detects the overflow and causes dhclient to exit with no security consequence. No security update for users of Red Hat Enterprise Linux 5 was needed.
• CVE-2009-1252 a buffer overflow flaw in NTP caught by FORTIFY_SOURCE. We issued an update as a remote attacker could still cause a denial of service.
• CVE-2009-0846, a uninitialized pointer free in krb5. glibc provides a hardened malloc/free implementation which mitigates the exploitability of this flaw, however we issued an update as a remote attacker could still cause a denial of service.

Previous updates

To compare these statistics with previous update releases we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the results illustrated by the following chart:

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.

MD2 signature verification

The first issue is that many web browsers still accept certificates with MD2 hash signatures, even though MD2 is no longer considered a cryptographically strong algorithm. This could make it easier for an attacker to create a malicious certificate that would be treated as trusted by a browser. It turns out that there are not many valid MD2 hash certificates around any more, and the main one that does exist is at the trusted root level anyway (and there is actually no need for a crypto library to verify the self-signature on a trusted root). So most vendors have chosen to address this issue by disabling MD2 completely for certificate verification. This is allocated CVE name CVE-2009-2409 ( single name for all affected products).

• OpenSSL. For upstream OpenSSL we have disabled MD2 support completely. This was done in two stages; the first was a patch in June 2009 that removed the redundant check of a trusted root self-signed certificate. Then in July, MD2 was totally disabled. So this issue does not affect OpenSSL 1.0.0 beta 3 or later. Although there have not yet been an upstream release of 0.9.8 containing this fix, a future OpenSSL 0.9.8 (after 0.9.8k) will disable MD2, probably in a few weeks.

• GnuTLS. The upstream GnuTLS library has for some time meant to have disabled MD2 support, although due to a broken patch it wasn't actually disabled correctly until January 2009. So this issue does not affect GnuTLS versions 2.6.4 and above, or GnuTLS versions 2.7.4 and above.

• NSS (and hence Firefox). The upstream NSS library since version 3.12.3 (April 2009) has disabled MD2 and MD4 by default (although legacy applications could turn it back on using an environment variable "NSS_ALLOW_WEAK_SIGNATURE_ALG" if they need to). Mozilla Firefox since version 3.5 has used this NSS version and therefore MD2 is disabled. I suspect this issue will get addressed in a future Firefox 3.0 update in the future too if they rebase to the new NSS.

There is no immediate panic to address this issue as a critical security issue, as in order for it to be exploited an attacker still has to create a MD2 collision with this root certificate; something that is as of today still a significant amount of effort.

My CVSS v2 base score for CVE-2009-2409 would be 2.6 (AV:N/AC:H/Au:N/C:N/I:P/A:N)

Differences in Common Name handling

This issue is about how Common Names are checked for validity by applications. For example if a server presents a certificate with two CN entries, how does the app validate those. Does it use the first one, the last one, or all of them?

• OpenSSL. OpenSSL provides an API that allow applications to check CN names any way they want. It turns out that, without sound guidance, applications have tended to do things differently. A summary of a few OpenSSL applications is in this Red Hat bugzilla comment. But as a CA should validate all CN names in a certificate being signing, these are really just bugs and do not have a security impact

Leading 0's in Common Name handling

The second issue is all about inconsistencies in the interpretation of subject x509 names in certificates. Specifically "issue 2b, subattack 1" is where a malicious certificate can contain leading 0's in the OID. The idea is that an attacker could add in some OID into a certificate that, when handled by the Certificate Authority, would appear to be some extension and ignored, but when handled by OpenSSL would appear to be the Common Name OID. So the attacker would present the certificate to a client application and it might think that the OID is actually a Common Name, and accept the certificate where it otherwise should not.

• OpenSSL. This is not a security issue for OpenSSL. Steve Henson explains: "OpenSSL does tolerate leading 0x80 but it does _not_ recognize this as commonName because the NID code checks for a precise match with the encoding. Attempts to print this out will never show commonName nor will attempts to look up using NID_commonName". However this will be addressed as a bug fix in the future.

• NSS (and hence Firefox). NSS is noted in the paper as having a similar issue, but again it's not fooled into treating the OID as a Common Name so this is not a security issue (and therefore I didn't check if this is already fixed in the new upstream NSS).

OID overflow in Common Name handling

"issue 2b, subattack 2" is where a malicious certificate can have a very large integer in the OID. The idea is that an attacker could add in some OID into a certificate that, when handled by the CA, would appear to be some extension and ignored, but when handled by OpenSSL would overflow and appear to be the Common Name OID. So the attacker would present the certificate to a client application using OpenSSL and it might think that the OID is actually a Common Name, and accept the certificate where it otherwise should not.

• OpenSSL. This issue was actually fixed upstream in September 2006 in OpenSSL 0.9.8d by switching to using the bignum library for handling the OID. Even for older versions though it's really not a security issue for the same reason as given earlier: the OpenSSL NID code checks for a precise match with the encoding. So attempts to print this out will never show it being a Common Name, nor will attempts to look it up as a Common Name succeed.

NULL bytes in Common Name handling

"issue 2, attack 2c" is regarding NULL terminators in a Common Name field. If an attacker is able to get a carefully-crafted certificate signed by a Certificate Authority trusted by a browser, the attacker could use the certificate during a man-in-the-middle attack and potentially confuse the browser into accepting it by mistake.

• NSS (and hence Firefox). This issue affected NSS and is assigned CVE-2009-2408. The upstream NSS library since version 3.12.3 (April 2009) has fixes to address this issue. Therefore Firefox 3.5 is not affected.

My CVSS v2 base score for CVE-2009-2408 would be 4.3 (AV:N/AC:M/Au:N/C:N/I:P/A:N)

OpenSSL 'compat mode' subject name injection

"issue 2d" is how the OpenSSL command line utility will output unescaped subject X509 lines to standard output. So if some utility runs the openssl application from the command line and parses the text output, and if an attacker can craft a malicious certificate in such a way they fool a CA into signing it, they could present it to the utility and possibly fool that utility into thinking fields were different to what they actually are, perhaps allowing the certificate to be accepted as legitimate.

• OpenSSL. This attack requires that some utility will parse the output of OpenSSL command line using the default 'compat' mode. Applications should never do this. Upstream OpenSSL are unlikely to address this issue directly, although in the future the default output mode perhaps could be changed to something other than 'compat', and it's likely a documentation update will remind users that parsing the output of running such an openssl command is not the right way to use OpenSSL.

OpenSSL ASN1 printing crash

• OpenSSL. This was reported by Dan to OpenSSL and fixed in March 2009. This was allocated CVE name CVE-2009-0590. Therefore OpenSSL 0.9.8k and later contains a fix for this issue.

My CVSS v2 base score for CVE-2009-0590 would be 2.6 (AV:N/AC:H/Au:N/C:N/I:N/A:P)

From time to time I publish metrics on vulnerabilities that affect Red Hat Enterprise Linux. One of the more interesting metrics looks at how far in advance we know about the vulnerabilities we fix, and from where we get that information. This post is abstracted from the upcoming "4 years of Enterprise Linux 4" risk report

For every fixed vulnerability across every package and every severity in Enterprise Linux 4 AS in the first 4 years of its life, we determined if the flaw was something we knew about a day or more in advance of it being publicly disclosed, and how we found out about the flaw.

For vulnerabilities which are already public when we first hear about them we still track the source as it's a useful internal indicator on where the security response team should focus their efforts.

So from this data, Red Hat knew about 51% of the security vulnerabilities that we fixed at least a day in advance of them being publicly disclosed. For those issues, the average notice was 21 calendar days, although the median was much lower, with half the private issues having advance notice of 9 days or less.

The chart below shows the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.2, up to and including the 5.3 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). So, for a given installation, the number of packages and vulnerabilities will probably be somewhere between the two.

So for a default install, from release of 5.2 up to and including 5.3, we shipped 45 advisories to address 127 vulnerabilities. 7 advisories were rated critical, 21 were important, and the remaining 17 were moderate and low.

For all packages, from release of 5.2 to and including 5.3, we shipped 61 advisories to address 181 vulnerabilities. 7 advisories were rated critical, 28 were important, and the remaining 26 were moderate and low.

The 7 critical advisories were for just 3 different packages:

1. Five updates to Firefox (July, July, September, November, December) where a malicious web site could potentially run arbitrary code as the user running Firefox. Given the nature of the flaws, ExecShield protections in RHEL5 should make exploiting these memory flaws harder.
2. An update to Samba (May), where a remote attacker who can connect and send a print request to a Samba server could cause a heap overflow. The Red Hat Security Response Team believes it would be hard to remotely exploit this issue to execute arbitrary code due to the default enabled SELinux targeted policy and the default enabled SELinux memory protection tests. We are not aware of any public exploit for this issue.
3. An update to OpenSSH (August), provided to mitigate an intrusion into certain Red Hat computer systems. The attacker was able to sign a small number of tampered packages but they were not distributed on the Red Hat Network. We classified this update as critical to ensure any tampered packages would be replaced with official packages.

Although not of critical severity, also of interest during this period were the spoofing attacks on DNS servers. We provided an update to BIND (July) adding source port randomization to help mitigate these attacks.

Updates to correct all of these critical vulnerabilities (as well as migitate the BIND issue) were available via Red Hat Network either the same day, or one calendar day after the issues were public.

In fact for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update available to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

To compare this with the last updates we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. For 5.2 to 5.3 there were two flaws blocked that would otherwise have required updates:

1. A double-free flaw in unzip. The glibc pointer checking limited the exploitability of this issue to just a crash of unzip, a client application, which does not have security implications. No security update was needed.
2. Two format string flaws in c++filt. The format string protection caused these issues to have no security implications. No security update was needed.

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

Actually this isn't surprising and is exactly what I'd expect; it's all down to third party applications.

Let's say you're browsing the web. It's more than likely that at some point you'll want to view some PDF files, watch some Flash content, or play a Java game. Those tasks are all dealt with by third party applications, although to the end user it's all part of the browser experience. Since your system is only as secure as its weakest link, you need to manage security updates for those third party applications just as carefully as you manage security updates for the rest of your system. That's why Adobe Reader, Java, Flash, and all the myriad of other applications you've installed in order to make your system useful have their own update mechanisms. Some applications on Windows will 'phone home' when they are run and check to see if they need to be updated, others deploy services that sit in the background looking for updates from time to time, others even check every time your system starts. Many don't get automated updates at all.

How do you deal with all that risk? I believe it's possible by providing an OS distribution which includes all the bits you'll likely need to make a useful computing environment, thereby taking away that update uncertainty. Red Hat ship several PDF viewers in our distributions for example, but we also ship (in an Extras channel) Adobe Reader. Our Security Response Team are monitoring for security issues in everything we ship, all the third party applications, and providing a single point of contact, a single notification system, and a single way to get the updates.

If Microsoft knew that say 25% of all their users installed Firefox, wouldn't they be better bundling it and providing their centralised automated updates for it, to reduce their customers overall risk? They do already bundle some third party applications, although it's been with mixed success as we found 3 years ago when they didn't provide security fixes for bundled Flash (ZDNet coverage).

This is, in part, why you've not seen me respond recently to the Vista security reports which compare vulnerability counts. In these reports they use a cut-down minimal Red Hat Enterprise Linux installation in order to make it look more like Windows for the comparisons. But this is completely backwards -- the fact that we're including and fixing the flaws using a common process in so much third party software is actually helping reduce the risk and protect real customers. For example we could easily cut our vulnerability count by shipping only one PDF viewer instead of four. But if we know that these other viewers are going to get installed by the customer anyway all we've done is to hide the vulnerability count elsewhere, and you've made the customers overall risk increase.

So it may seem counter-intuitive but we should ship as much third party applications (that we know people use) as we can, because a single managed security update and notification process will decrease a users overall risk. The fewer third party applications that users have to get from elsewhere and install and manage for themselves the better in my opinion.

There are plenty of new products to come, so we're currently hiring for another engineer to join the response team. The full job details are here:

https://redhat.ats.hrsmart.com/cgi-bin/a/highlightjob.cgi?jobid=3685

Although the location is specified as the Czech Republic there is actually no specific restriction on the location of this position, and the candidate could be located at any one of the world-wide Red Hat offices, or potentially even remote.

If you are interested please submit your resume through the online application process, or feel free to mail me with questions.

CVSS v2 Base Score 6.5 (AV:N/AC:L/Au:S/C:P/I:P/A:P)

This is really a moderate severity flaw because you need a remote attacker who has the ability to start/stop/control ZoneMinder, and you really should protect your ZoneMinder installation so you don't allow arbitrary people to control your security system. (Although I think at least one distributor package of ZoneMinder doesn't protect it by default, and you can find a few unprotected ZoneMinder consoles using a web search).

I discovered this because when we went on holiday early in April I forgot to turn down the heating in the house. Our heating system is controlled by computer and you can change the settings locally by talking to a Jabber heating bot (Figure 1). But remotely over the internet it's pretty locked down and the only thing we can access is the installation of ZoneMinder. So without remote shell access, and with an hour to spare at Heathrow waiting for the connecting flight to Phoenix, I figured the easiest way to correct the temperature was to find a security flaw in ZoneMinder and exploit it. The fallback plan was to explain to our house-minder how to change it locally, but that didn't seem as much fun.

So I downloaded ZoneMinder and took a look at the source. ZoneMinder is a mixture of C and PHP, and a few years ago I found a buffer overflow in one of the C CGI scripts, but as I use Red Hat Enterprise Linux exploiting any new buffer overflow with my ZoneMinder compiled as PIE definately wouldn't be feasible with just an hours work. My PHP and Apache were up to date too. So I focussed on the PHP scripts.

A quick grep of the PHP scripts packaged with ZoneMinder found a few cases where the arguments passed to PHP exec() were not escaped. One of them was really straightforward to exploit, and with a carefully crafted URL (and if you have authorization to a ZoneMinder installation) you can run arbitrary shell code as the Apache httpd user. So with the help of an inserted semicolon and one reverse shell I had the ability to remotely turn down the heating, and was happy.

I notified the ZoneMinder author and the various vendors shortly after and updates were released today (a patch is also available)

Figure 1: Local heating control

My full risk report was published yesterday in Red Hat Magazine and reveals the state of security since the release of Red Hat Enterprise Linux 4 including metrics, key vulnerabilities, and the most common ways users were affected by security issues.

"Red Hat knew about 49% of the security vulnerabilities that we fixed in advance of them being publicly disclosed. For those issues, the average notice was 21 calendar days, although the median was much lower, with half the private issues having advance notice of 8 days or less."

First thing to do was to find the original source of the advisory, as co-ordination centres and research firms are known to often play the Telephone game, with advisory texts mangled beyond recognition. Following the links led to the actual advisory on the HP site. This describes the vulnerability as follows:

But then they give the CVE name for the flaw, CVE-2007-6388, which is a known public flaw fixed last month in various Apache versions from the ASF and in updates from various vendors that ship Apache (including Red Hat).

This flaw is a cross-site scripting flaw in the mod_status module. Note that the server-status page is not enabled by default and it is best practice to not make this publicly available. I wrote mod_status over 12 years ago and so I know that this flaw is exactly how the ASF describes it; it definitely can't let a remote attacker execute arbitrary code on your Apache HTTP server, under any circumstances.

I fired off a quick email to a couple of contacts in the HP security team and they confirmed that the flaw they fixed is just the cross-site scripting flaw, not a remote code flaw. The CVSS ratings they give in their advisory are consistent with it being a cross-site scripting flaw too.

So happy with a false alarm we cancelled our Critical Action Plan and I went off and had a nice weekend practicing taking panoramas without a tripod ready for an upcoming holiday. My first attempt came out better than I expected:

Hi! I'm Mark Cox. This blog gives my thoughts and opinions on my security work, open source, fedora, home automation, and other topics.

pics from my twitter:

popular tags: [all], apache, apachecon, apacheweek, cve, cvss, fedora, financial, geocaching, ha, metrics, microsoft, nashville, north carolina, red hat summit, redhat, security, trips