Late last month I spent a day with the Red Hat Magazine team talking
about vulnerability response. The first video
is now available
and talks about the role of Red Hat in dealing
with vulnerabilities in third party software. The video was shot in
my home office which explains the calming green paint; it's hard to
get too stressed in a pale green room.
Red Hat Enterprise Linux 5.1 was released today, around 8 months since the
release of 5.0 in March 2007. So let's use this opportunity to take a quick
look back over the vulnerabilities and security updates we've made in that time,
specifically for Red Hat Enterprise Linux 5 Server.
The graph below shows the total number of security updates issued for Red Hat
Enterprise Linux 5 Server up to and including the 5.1 release,
broken down by severity. I've split it into two columns, one for the packages
you'd get if you did a default install, and the other if you installed every
single package (which is unlikely as it would involve a bit of manual effort
to select every one). So, for a given installation, the number
of packages and vulnerabilities will be somewhere between the two extremes.
So for all packages, from release up to and including 5.1, we shipped 94 updates
to address 218 vulnerabilities. 7 advisories were rated critical, 36 were
important, and the remaining 51 were moderate and low.
For a default install, from release up to and including 5.1, we shipped 60
updates to address 135 vulnerabilities. 7 advisories were rated critical, 26
were important, and the remaining 27 were moderate and low.
- These figures include ten updates we released on the day we shipped 5.0. This was
because we froze package updates some months before releasing the product. Only
one of those updates was rated critical, an update to Firefox.
- The six other critical updates were:
- Three more updates to Firefox (May, July, October)
where a malicious web site could potentially run arbitrary code as the
user running Firefox. Given the nature of the flaws, ExecShield
protections in RHEL5 should make exploiting these memory flaws
- An update to the Kerberos telnet deamon (April)
A remote attacker who can access the telnet
port of a target machine could log in as root without requiring a
password. None of the standard protection mechanisms help prevent
exploitation of this issue, however the krb5 telnet daemon is not
enabled by default in Enterprise Linux 5 and the default firewall rules
block remote access to the telnet port. This flaw did not affect the
more common telnet daemon distributed in the telnet-server
- An update to Samba (May) where
a remote attacker could cause a heap overflow. In addition to
ExecShield making this harder to exploit, the impact of any sucessful
exploit would be reduced as Samba is constrained by an SELinux targeted
policy (enabled by default).
- An update to the PCRE library (November). This
was labelled critical because the Konqueror web browser uses PCRE to handle
site in Konqueror could trigger this issue. (Konqueror is not part of
a default install, but I've left this issue as critical in the results).
- Updates to correct all of these critical issues were available via Red Hat
Network within a day of the issues being public.
Red Hat Enterprise Linux 5 shipped with a number of security technologies
designed to make it harder to exploit vulnerabilities and in some cases block
exploits for certain flaw types completely. For the period of this study there
were two flaws blocked that would otherwise have required critical updates:
- A stack buffer overflow flaw in the RPC library in Kerberos.
This flaw was blocked by FORTIFY_SOURCE which removed the possibility of remote
code execution. We still issued an update,
as a remote attacker could trigger this flaw and cause Kerberos to crash.
- Another flaw in Kerberos, this time due to the free of an invalid
pointer. This flaw was blocked by glibc, although a remote attacker could still
a crash, so we
issued an update.
This data is interesting to get a feel for the risk of running Enterprise Linux
5 Server, but isn't really useful for comparisons with other versions or
distributions -- for example, a default install of Red Hat Enterprise 4AS did
not include Firefox. You can get the results I presented above for yourself by
using our public security
measurement data and tools, and run your own metrics for any given Red Hat
product, package set, timescales, and severities.
Although Red Hat is well known for Red Hat Enterprise Linux we actually have a large number of other supported products, both layered on top of Enterprise Linux (like Red Hat Application Stack) and stand-alone (like Red Hat Directory Server). The majority of these products are serviced through the Red Hat Network and get our security advisories in a standard way and are included in the Security Response Team metrics
. But our analysis scripts were not particularly consistent in dealing with product names.
Common Platform Enumeration (CPE) is a naming scheme designed to combat these inconsistencies, and is part of the 'making security measurable' initiative from Mitre. From today we're supporting CPE in our Security Response Team metrics: we publish a mapping of Red Hat advisories to both CVE and CPE platforms (updated daily) and you can use CPE to filter the metrics. Some examples of CPE names:
cpe://redhat:enterprise_linux:5:server/firefox -- the Firefox browser package on Red Hat Enterprise Linux 5 server.
cpe://redhat:enterprise_linux:3 -- Red Hat Enterprise Linux 3
cpe://redhat/xpdf -- the xpdf package in any Red Hat
cpe://redhat:rhel_application_stack:1 -- Red Hat Application Stack
For the past 12 months I've been keeping metrics on the types of issues that get
reported to the private Apache Software Foundation security alert
email address. Here's the summary for Jul 2006-Jun 2007 based
on 154 reports:
User reports a security vulnerability|
(this includes things
later found not to be vulnerabilities)
User is confused because they visited a site "powered by Apache"|
(happens a lot when some phishing or spam points to a site that is
taken down and replaced with the default Apache httpd page)
User asks a general product support question|
User asks a question about old security vulnerabilities|
User reports being compromised, although non-ASF software was at fault|
(For example through PHP, CGI, other web applications)
That last one is worth restating: in the last 12 months no one who
contacted the ASF security team reported a compromise that was
found to be caused by ASF software.
The National Vulnerability Database provides a public severity rating
for all CVE named vulnerabilities, "Low" "Medium" and "High",
which they generate automatically based on the CVSS score their
analysts calculate for each issue. I've been interested for some time to see
how well those map to the severity ratings that Red Hat give to
issues. We use the same ratings
and methodology as Microsoft and others use, assigning "Critical"
for things that have the ability to be remotely exploited automatically
through "Important", "Moderate", to "Low".
Given a thundery Sunday afternoon I took the last 12 months of all possible
vulnerabilities affecting Red Hat Enterprise Linux 4 (from 126 advisories across
all components) from my metrics page and compared to NVD using their provided XML
data files. The result broke down like this:
So that looked okay on the surface; but the diagram above implies that
all the issues Red Hat rated as Critical got mapped in NVD to High. But
that's not actually the case, and when you
look at the breakdown you get this result: (in number of vulnerabilities)
That shows nearly half of the issues that NVD rated as High actually only
affected Red Hat with Moderate or Low severity. Given our policy is to fix the
things that are Critical and Important the fastest (and we have a pretty impressive record
for fixing critical issues), it's no wonder that recent vulnerability studies
that use the NVD mapping when analysing Red Hat vulnerabilities have some
significant data errors.
I wasn't actually surprised that there are so many differences: my
hypothesis is that many of the errors are due to the nature of how
vulnerabilities affect open source
software. Take for example the Apache HTTP server. Lots of companies ship
Apache in their products, but all ship different versions with different
defaults on different operating systems for different architecture compiled with
different compilers using different compiler options. Many Apache
vulnerabilities over the years have affected different platforms in
ways. We've seen an Apache vulnerability that leads to arbitrary code execution
on older FreeBSD, that causes a denial of service on Windows, but that was
unexploitable on Linux for example. But it has a single CVE identifier.
So if you're using a version of the Apache web server you
got with your Red Hat Enterprise Linux distribution then you need to
rely on Red Hat to tell you how the issue affects the version they
gave you -- in the same way you rely on them to give you an update
to correct the issue.
I did also spot a few instances where the CVSS score for a given vulnerability
was not correctly coded. CVSS version 2 was released last week and once NVD is
based on the new version I'll redo this analysis and spend more time submitting
corrections to any obvious mistakes.
But in summary: for multi-vendor software the severity rating for a given
vulnerability may very well be different for each vendors version. This is a
level of detail that vulnerability databases such as NVD don't currently
capture; so you need to be careful if you are relying on the accuracy of
third party severity ratings.
It sometimes seems like me and my team are pushing security updates every day, but actually a default installation of Enterprise Linux 4 AS was vulnerable to only 3 critical security issues in the first two years since release. But to get a picture of the risk you need to do more than count vulnerabilities. My full risk report
was published today in Red Hat Magazine and reveals the state of security since the release of Red Hat Enterprise Linux 4 including metrics, key vulnerabilities, and the most common ways users were affected by security issues. It's all about transparency, highlighting the bad along with the good, and rather than just giving statistics and headlines you can game using carefully selected initial conditions we also make all our raw data available too so we can be held accountable.
Red Hat Enterprise Linux 5 was released back in March 2007 so
let's take a quick look back over the first three months of security
updates to the Server distribution:
- We released updates to ten packages on the day we shipped the
product. These is because we freeze packages some months before releasing
the product (more
information about this policy). Only one of those updates was rated
critical, an update to Firefox.
- For the three months following release we shipped 31 more advisories to
address 56 vulnerabilities: 3 advisories were rated critical, 8 were
important, and the remaining 20 were moderate and low.
- The three critical advisories were:
update to Firefox where a malicious web site could potentially
run arbitrary code as the user running Firefox. Given the nature of the
flaws, Execshield protections in RHEL5 should make exploiting
these issues harder.
An update to Samba where a remote attacker could cause a heap overflow.
In addition to Execshield making this harder to exploit, the impact of
any sucessful exploit would be reduced as
Samba is constrained by an SELinux targeted policy (enabled by default).
An update to the Kerberos telnet deamon. A remote attacker who can access the
telnet port of a target machine could
log in as root without requiring
a password. None of the standard protection mechanisms help prevent
exploitation of this issue, however the krb5 telnet daemon is not
enabled by default in RHEL5 and the default firewall rules
block remote access to the telnet port. This flaw did not affect the
more common telnet daemon distributed in the telnet-server package.
Updates to correct all of these critical issues were available via
Red Hat Network on the same day as the issues were made public.
This data is interesting to get a feel for the risk of running EL5, but isn't
really useful for comparisons with other versions or distributions -- for
example previous versions didn't include Firefox in a default Server
As part of our security measurement work since March 2005 we've been tracking
how the Red Hat Security Response Team first found out about each vulnerability
we fix. This information is interesting as can show us which
relationships matter the most, and identify trends in vulnerability disclosure.
So for two years to March 2007 we get the following results (in percent):
I've separated the bars into two sections; the red sections are where
we get notice of a security issue in advance of it being public (where
we are told about the issue 'under embargo'). The grey sections
are where we are reacting to issues that are already public.
The number of issues through researchers and co-ordination centers seem lower
than perhaps expected, this is because in many cases the researcher will tell a
group such as vendor-sec rather than each distributor separately, or the
upstream project directly.
Red Hat has shipped products with randomization, stack protection, and other security mechanisms turned on by default since 2003. Vista recently shipped with similar protections and I read today an article
about how the Microsoft Security Response Team were not treating Vista any differently when rating the severity of security issues.
The Red Hat Security Response team use a similar
guide for classification and I thought it would be worth clarifying how we handle this very situation.
We rate the impact of individual vulnerabilities on the same four point scale, designed to be an at-a-glance guide to how worried Red Hat is about each security issue. The scale takes into account the potential risk of a flaw based on a technical analysis of the exact flaw and it's type, but not the current threat level. Therefore the rating given to an issue will not change if an exploit or worm is later released for a flaw, or if one is available before release of a fix.
For the purpose of evaluating severities, our protection technologies fall into
roughly three categories:
- Security innovations that completely block a particular type of security
flaw. An example of this is Fortify
Source. Given a particular vulnerability we can evaluate if the flaw would
be caught at either compile time or run time and blocked. Because this is
deterministic, we will adjust the security severity of an issue where we can
prove it would not be exploitable.
- Security innovations that should block a particular security flaw from being
remotely exploitable. Examples of this would be support for NX, Randomisation,
and Stack canaries.
Although these technologies can reduce the likelyhood of exploiting certain
types of vulnerabilities, we don't take them into account and don't downgrade
the security severity.
- Security innovations that try to contain an exploit for a vulnerability.
I'm thinking here of SELinux. An attacker who can exploit a flaw in any of the
remotely accessible daemons protected by a default SELinux policy will find
themselves tightly constrained by that policy. We do not take SELinux into
account when setting the security severity.
I've not been keeping a list of vulnerabilities that are deterministically
blocked, but I have a couple of examples I recall where we did alter the
- In October 2005 a buffer overflow flaw
was found in the authentication
code of wget. On Fedora Core 4 this flaw was not exploitable it was
caught by Fortify Source.
- In August 2004, a double-free flaw in
MIT Kerberos KDC was announced.
For users of Red Hat
Enterprise Linux 4 users we were able to downgrade the severity of
this flaw from critical as glibc hardening
prevented this double-free flaws being exploited. The flaw was
rated important as it could still lead to a remote denial of service
Enterprise Linux 5 has been consuming much of my time over the last few months. From work on the signing server
and new key policy, through testing of the new update mechanism, and continuing audits of outstanding
vulnerabilities. Yesterday was release day, and we also pushed security updates for 12 packages in Enterprise Linux 5.
It may seem surprising that we release security updates for a product exactly at the same time we release it, but product development is frozen for some weeks before we release the product to give time testing from the various Quality Engineering teams as well as release engineering
work. During that time we want to minimise the number of changes that will invalidate the overall testing, so we instead prepare the changes as updates. Since the vulnerabilities being fixed are already public, we push the updates out as soon as we can; holding them off to some scheduled monthly date would just increase customer risk.
Security advisories for Enterprise Linux 5 are available from the usual places, on the web, sent to the enterprise-watch-list mailing list, and via OVAL definitions. Red Hat Network subscribers can also get customized mails for the subset of issues that affect the packages they actually have installed.
For me, what's going to be interesting to watch over the next few months is how specific vulnerabilities and exploits affect this platform. Red Hat Enterprise Linux 5 packages are compiled both with
Fortify Source and stack
smashing protection in addition to all the security features that were in version 4. I'll be reporting on what difference this makes through the year.