Last night I wrote an email advising a client about the Heartbleed vulnerability and its impact on operations. I’m posting that here, as it’s a little more balanced and professional (and better written) than the blog entry I wrote last night. Of course I’ve omitted names to protect my clients’ privacy. (And I asked permission before publishing this.)
On April 7, 2014, the OpenSSL project released a security advisory for a vulnerability called CVE-2014-0160 (aka “Heartbleed”).
- https://www.openssl.org/news/secadv_20140407.txt
- https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-0160
- A great writeup from some of the researchers who discovered the vulnerability can be found at http://heartbleed.com/
This is a newly-announced vulnerability. The following is based on our reading of the security advisory and related information. Further information on this topic is becoming available rapidly, and we will update you as we get new facts and as our understanding of the facts changes.
OpenSSL is the tool that provides secure connections to websites. It’s what makes https:// connections work properly. Our project uses OpenSSL to secure the public’s connections to our web servers.
The vulnerability allows an attacker to send a specially-crafted packet to OpenSSL, and receive a dump of up to 64KB of the server’s memory. Because the attacker can send multiple packets each requesting a different 64KB chunk of the server’s memory, in effect the attacker can read the entire memory (RAM) of the server.
The vulnerable versions of OpenSSL (1.0.1 and 1.0.2) were released in March 2012, and all versions released between then and today are vulnerable to this attack. These versions are very widely-distributed. (Including on Ubuntu 12.04 Server, the operating system we use for the our servers.)
Because this is an attack that a very large number of servers on the Internet are vulnerable to, it is likely that this attack will be quickly “weaponized” so that unskilled attackers can use it in an automated fashion. At present, only highly-skilled attackers can use it.
We have verified using http://filippo.io/Heartbleed/ that our project is vulnerable to this attack.
All of this lends a great deal of urgency to the update and remediation process:
- The very first step is to update the version of OpenSSL on our servers to the latest version, released today. This can be done quickly and easily, and should be done as soon as possible. It is extremely likely that there will be no service interruption (other than on each server as it is updated) resulting from this update.
- Once the servers have been updated, they are no longer vulnerable to this attack. However, attackers could be in possession of private information that they copied off the server before OpenSSL was updated. (It appears that security researchers have been aware of this issue since December 2013, but attackers could have been using this attack any time from March 2012 to the present.)
- The following information should definitely be revoked and reissued:
- SSL Certificates
- VPN Certificates (VPN also uses SSL)
Until these certificates are revoked and reissued, we have to assume that attackers can eavesdrop on and modify the HTTPS/SSL traffic of our customers reaching our website, and of our VPN traffic.
Anything that has been in RAM on the server, even briefly, should be revoked and reissued. Our developers will have to study the issue to determine what needs to be revoked. However, a partial list of things that may have been in memory and may have been compromised includes:
- End-user usernames and passwords
- API keys
- Passwords and authentication certificates to internal and external services, including our databases and monitoring services.
We are not aware at this time if any Intrusion Detection System or network monitoring can detect this attack. This attack does not show up in OpenSSL or system logs.
