Since 1999, all RPM packages in Red Hat products have been gpg signed by the master key "Red Hat, Inc <firstname.lastname@example.org>" (keyid DB42A60E). I'll call this the legacy signing key for the rest of this article. This signature is one of two security mechanisms we use to ensure that customers can trust the installation of packages and their updates. The other is that the update client, up2date, checks the SSL server signature when it connects to the Red Hat Network to ensure that it only talks to official Red Hat servers; so removing the possibility of a man-in-the-middle attack.
From 2007, all new products will be signed with a different master key, "Red Hat, Inc. (release key) <email@example.com>" (keyid 37017186). This includes Red Hat Enterprise Linux 5, and any other new products that use RPM packages. The exception to this rule is that any new layered products designed for older versions of Enterprise Linux will still use the legacy key: so for example, a new version of the Application Stack for Red Hat Enterprise Linux 4 will be signed with the legacy key.
The legacy key hasn't been compromised so why change keys? It's all to do with the way the keys are stored and managed internal to Red Hat. The legacy key is a software key and so the key material exists, protected by a passphrase, on a hard disk. When packages need to be signed one of the Red Hat authorised signers manually runs a signing command, this calls
rpm --resign which asks for the passphrase then in turn calls out to GNUpg to do the actual signature creation. So the authorised signers not only had the ability to sign with the key, but they also have the ability to read the key material. In theory this means that a malicious internal signer could copy the key, take it away with them, and sign whatever and whenever they
wanted. Or, more likely, a clever intruder who gained access to our internal network could perhaps capture the key and passphrase, compromising the key. The risks mean we've had to be really careful who has signing privileges with the legacy key and how the key signing is handled.
The new key, in contrast, was created in a hardware cryptographic device which does not allow the unprotected key material to be exported. This means we can give authorised signers the ability to sign with the key, but no one can ever can get access to the key material itself. This is an important distinction. If for example a current authorised signer switches roles and is no longer responsible for package signing we can instantly revoke their rights and know that they no longer have the ability to sign any more packages with that key.
There was no off-the-shelf solution available for hardware-based RPM key management, so we developed one internally ourselves. We used nCipher nShield hardware security modules (FIPS 140-2 validated) for the key protection along with custom patches I developed to interface RPM/GNUpg to the unit. At the same time we also introduced an extra layer of abstraction to the signing software, so we can authorize signers using their existing internal kerberos credentials.
So, as a customer, you won't really notice any difference. For Red Hat Enterprise Linux 5 you'll find the public keys on our website as well as in the
/etc/pki/rpm-gpg/ directory and you'll be prompted when updating or installing new packages for the first time to import that new public key.
This change basically makes it easier for us to protect our signing key and reduce the risk of it being compromised, therefore reducing the chances we'll need to change the key and involve customer effort in the future.
|Fedora Core||Red Hat Enterprise Linux|
|Default requires signed updates||Y||Y||Y||Y||Y||Y||Y||Y|
|NX emulation using segment limits by default||Y||Y||Y||Y||Y||Y||since 2004Sep||Y|
|Support for Position Independent Executables (PIE)||Y||Y||Y||Y||Y||Y||since 2004Sep||Y|
|ASLR for Stack/mmap by default||Y||Y||Y||Y||Y||Y||since 2004Sep||Y|
|ASLR for vDSO (if vDSO enabled)||no vDSO||Y||Y||Y||Y||Y||no vDSO||Y|
|Restricted access to kernel memory by default||Y||Y||Y||Y||Y||Y|
|NX by default for supported processors/kernels||since 2004Jun||Y||Y||Y||Y||since 2004Sep||Y|
|Support for SELinux||Y||Y||Y||Y||Y||Y|
|SELinux default enabled with targetted policies||Y||Y||Y||Y||Y|
|glibc heap/memory checks by default||Y||Y||Y||Y||Y|
|Support for FORTIFY_SOURCE, used on selected packages||Y||Y||Y||Y||Y|
|All packages compiled using FORTIFY_SOURCE||Y||Y||Y|
|Support for ELF Data Hardening||Y||Y||Y||Y|
|All packages compiled with stack smashing protection||Y||Y|
|OVAL compatible||since 2006May||since 2006May|
First, the Vulnerability Type Distribution graph. This is not really a big surprise, the most common vulnerabilities we fix are buffer overflows. Technologies such as ExecShield (PIE, support for NX, FORTIFY_SOURCE and so on) were designed specifically to reduce the risk of being able to exploit this flaw type. Secondly, compared to the industry as a whole we fix far less web application flaws such as cross-site scripting or SQL injection. This result is to be expected as most of these are in PHP web applications we don't ship in our distributions.
This issue is technically interesting as the off-by-one only lets you write one pointer to the space immediately after a stack buffer. So the ability to exploit this issue is totally dependent on the stack layout for a particular compiled version of mod_rewrite. If the compiler used has added padding to the stack immediately after the buffer being overwritten, this issue can not be exploited, and Apache httpd will continue operating normally. Many older (up to a year or so ago) versions of gcc pad stack buffers on most architectures.
The Red Hat Security Response Team analysed Red Hat Enterprise Linux 3 and Red Hat Enterprise Linux 4 binaries for all architectures as shipped by Red Hat and determined that these versions cannot be exploited. We therefore do not plan on providing updates for this issue.
In contrast, our Fedora Core 4 and 5 builds are vulnerable as the compiler version used adds no stack padding. For these builds, the pointer being overwritten overwrites a saved register and, unfortunately, one that has possible security consequences. It's still quite unlikely we'll see a worm appear for this issue that affects Fedora though: for one thing, the vulnerability can only be exploited when mod_rewrite is enabled and a specific style of RewriteRule is used. So it's likely to be different on every vulnerable site (unless someone has some third party product that relies on some vulnerable rewrite rules). Even then, you still need to be able to defeat the Fedora Core randomization to be able to reliably do anything interesting with this flaw.
So, as you can probably tell, I spent a few days this week analysing assembler dumps of our Apache binaries on some architectures. It was more fun than expected; mostly because I used to code full-time in assembler, although that was over 15 years ago.
In the past I've posted timelines of when we found out about issues and dealt with them in Apache; so for those who are interested:
20060721-23:29 Mark Dowd forwards details of issue to firstname.lastname@example.org 20060722-07:42 Initial response from Apache security team 20060722-08:14 Investigation, testing, and patches created 20060724-19:04 Negotiated release date with reporter 20060725-10:00 Notified NISCC and CERT to give vendors heads up 20060727-17:00 Fixes committed publically 20060727-23:30 Updates released to Apache site 20060828 Public announcement from Apache, McAfee, CERT, NISCCHere is the patch against 2.0, the patch against 1.3 or 2.2 is almost identical.
This vulnerability does not affect Red Hat Enterprise Linux 2.1 or 3 as they are based on 2.4 kernels.
Red Hat Enterprise Linux 4, Fedora Core 4, and Fedora Core 5 do not support the a.out binary format, causing the exploit to fail. We are not currently aware of any way to exploit this vulnerability if a.out binary format is not enabled. In addition, a default installation of these OS enables SELinux in enforcing mode. SELinux also completely blocks attempts to exploit this issue.
For more technical details of this issue please see bz#198973
The Red Hat Security response team have therefore rated this as having moderate security severity for Enterprise Linux 4. No asynchronous kernel update for this issue is currently planned; the fix for the flaw will be included in some later scheduled update.
One of the goals of Red Hat Enterprise Linux is to maintain backward compatibility of the packages we ship where possible. This goal means making sure that when we release security updates to fix vulnerabilities that we include just the security fixes in isolation, a process known as backporting. Backporting security fixes has the advantage that it makes installing updates safer and easier for customers, but has the disadvantage that it can cause confusion to people unfamiliar with the process who try to use the version number of a particular piece of software to determine it's patch status.
In 2002, Red Hat started publishing Common Vulnerability and Exposures (CVE) vulnerability identifiers on every security advisory in order to make it easy to see what we fixed and how. Customers need only know the CVE identifiers for the vulnerabilities they are interested in and can then find out quickly and easily which of our updates addressed that particular vulnerability. CVE is now used on security advisories from nearly all the major vendors.
Red Hat has a single common mechanism for keeping systems up to date with security errata, the Red Hat Network. The Red Hat Network looks at a customers machines to determine which updates are required and gives anything from a customised notification that an update is available through to automated installation. Third party patch auditing tools don't have such an easy time figuring out what up dates are required: they have to maintain their own list of Red Hat package versions against vulnerability names. As this list is different for each operating system version from each potential vendors, these tools are prone to many errors and lag behind our updates.
We've also found customers that query the Red Hat Network errata pages directly to gather information about our security advisories and put them into a format they can integrate with their own processes. Many customers take feeds of vulnerability data, usually in some XML format, from third party security vulnerability companies.
MITRE recognised both of these issues a number of years ago when they founded the Open Vulnerability and Assessment Language project, OVAL in 2002. The aim of OVAL is to provide a language for defining how to test for vulnerabilities and system configuration errors in an open and cross-platform manner. Red Hat was a founding board member of the OVAL project as part of our overall commitment to security quality.
So Red Hat now publishes OVAL 5 definitions for our Red Hat Enterprise Linux 3 and 4 security advisories. Each security advisory gets a separate XML OVAL file which defines the steps needed to test if an update is required for the target system. In an ideal world every Red Hat Enterprise Linux system would be consuming every update from Red Hat Network automatically, but for those that don't or where systems have been disconnected for some time, these definitions can help determine the patch status. In addition, these definitions contain selected info rmation from our advisories which can be combined with vulnerability feeds from third parties.
Red Hat OVAL patch definitions contain:
The actual OVAL definitions themselves are available from http://www.redhat.com/oval/ and are public within a couple of hours of an advisory being pushed to the Red Hat Network. OVAL definitions for all previous Red Hat Enterprise Linux 3 and 4 advisories are also available. At present we do not ship OVAL tools such as a definition interpreter, but there are severalopen-source and commercial OVAL-compatible tools available.
For the future we encourage other vendors to publish definitive OVAL definitions for their products too, and we hope to work on compatibility testing with other operating system and tool vendors.
More information about the make-up of the OVAL patch definitions can be found at the MITRE OVAL site. An FAQ about our implementation and where to contact us with comments or questions is also available.
American Airlines Experiences Suck
More American Airlines Experiences Suck More
First was the debacle which was a 7 hour delay getting to Nashville after a flight was cancelled. Now, at 9pm the day before my 6am flight from Nashville to London tommorrow they cancel my flight and are unable to get me to London in time for my Monday meeting. So I miss my meeting and total for the week I'll have had 18 hours of delays. Although perhaps I shouldn't blog this until I'm home as I'm still in Nashville and, nice that it is, don't want to spend another year here. So thats four out of my last six AA trips that have gone significantly wrong, and I only used AA this time because I wanted to upgrade and had miles left.
However, rant aside, this trip was all about the Red Hat Summit. I was pleasantly suprised by how smoothly it ran and how useful it was to have face-to-face meetings with some of the people I interact with daily by computer. There's a few cool things that the trip acted as a catalyst for, but you'll need to wait to find out ;) I tried to speak to many different attendees over breaks in the days, and consensus was positive with all the first-timers wishing to attend again in the future.