Photo by Inbetween Architects on Unsplash

What The x.509 Is Wrong With Gmail?


You don’t often stop to think about x.509 and the Public Key Infrastructure (PKI) that authenticates our Internet connections. Allow me to explain why you should.

Transport Layer Security (TLS) uses x.509 certificates to authenticate connections. In your every-day use of the Internet, this means that you get a certificate from a server when you connect over HTTPS (for example.) This certificate is the only reasonable means you have to verify the identity of a server.

Why does this matter? I’m glad you asked.

Just because a web site looks like your bank’s web site does not mean that it is. The same is true of any server. Plenty of people would love to have you enter your Gmail password, credit card information, or social security number on a web site that is not what you think. In this past this was called wire fraud. Now it is called identity theft.

There is one other reason that has become public recently. At least one agency of the United States government has been said to perform man-in-the-middle attacks against their targets. This means that they intercept connections. They do this by responding to a web browser’s request faster than the website from which the require was made. When the request uses HTTPS, the attacker must also present a valid certificate.

I explained that we get a certificate when we connect to a server using HTTP. However, in order to trust this certificate in the traditional sense, you must trust the organization that signed the certificate (called the certificate authority or CA.)

In my installation of Firefox 25.0.1, there are more than 90 trusted certificate authorities. Do I really trust this huge list of organizations to validate the identity of every site I visit? The answer varies from person to person and organization to organization. I might trust GTE Corporation more than I trust, Inc. yet, my browser is configured to trust each one completely. (Some of these certificate authorities have also been accused of signing fake certificates on behalf of government agencies.)

Is There a Solution?

From a security perspective, it would ideal if your bank gave you a copy of their certificate when you went to the bank in person. Then you would have no need to trust a certificate authority. (This is true of any public key authentication scheme.) But this doesn’t solve the problem for Gmail or eBay.

Google has tried to provide a partial solution for Google Chrome users by pinning certificates. (OWASP has a wiki page on the topic.) This is nice for Google Chrome users, but doesn’t help if someone attacks your Android phone’s connection to an IMAPS server.

The Perspectives project was started by Dan Wendlandt at CMU. The project is an attempt to use the the concept of notary servers scattered around the Internet to detect man-in-the-middle attacks. It has an easy-to-use Firefox extension and provides a Python-based notary server so anyone can run a notary.

The TACK project by Moxie Marlinspike is Moxie’s most recent attempt to fix our current PKI. (His previous attempt—the now defunct—was similar to the Perspectives Project.) TACK stands for Trust Assertions for Certificate Keys. The project draft “defines a TLS Extension that enables a TLS server to support ‘pinning’ to a self-chosen signing key.” In English, the project adds a layer of indirection between the CA and the server’s certificate to protect against rogue or compromised CAs.

Finally, several organizations, including EFF (SSL Observatory) and GRC (Fingerprints) have projects which collect information about certificates being served by various sites.

What about Gmail?

I’ve been a Gmail user for more than five years. Only recently have I decided that I needed to evaluate the security trade-offs of using Gmail (and several other web services.) As part of that, I started using the Perspectives Project’s Firefox plugin and notary server. When I started to observe the certificates coming back from Gmail, I noticed that the certificate my browser receives never matches the certificates observed by Perspective notaries—even the notary server running on the same system as my browser.

In this video, you can see that something very unusual happens when I connect to with a web browser as opposed to using OpenSSL. (This is true for several other Google services like Instead of getting the same certificate (as identified by the certificate’s fingerprint,) the two connections get different certificates.

You can try it for yourself from a shell, provided you have OpenSSL installed. Use this command line. Don’t forget to press CTRL+D to close the connection:

$ openssl s_client -connect -showcerts | openssl x509 -fingerprint -md5

I have not observed this behavior on any other sites or services on the Internet. Initially, I thought someone was attacking my Gmail connection. I have since observed the same behavior whether I connect from my home, my Verizon wireless hotspot, or free public WiFi in an airport. I’ve also checked with another tech-savvy Gmail user who lives in a different city and reports the same behavior.

This leaves two possible causes. The first and more likely cause is that some of Google’s servers send back different certificates depending on the TLS request sent by the browser or OpenSSL command line tool. Alternately, some party between Google and my laptop intercepts the connections and provides my browser with a different but valid certificate. It seems far more likely that Google is returning different certificates, specifically in pursuit of forward secrecy.

In fact, Google wrote about their work two years ago. In my shock and excitement, I had failed to check any Google sites in Safari or another browser that doesn’t support forward secrecy (in conjunction with the ciphers accepted by Google.) If I had, I would have noticed that other browsers get the same certificate as the OpenSSL command line tool.

Have you seen this behavior on other sites? Did I reach the wrong conclusion? Let me know in the comments, on Twitter or send me e-mail (tfarrell at this domain; PGP key D79453F8).

+ more

Accurate Timing

Accurate Timing

In many tasks we need to do something at given intervals of time. The most obvious ways may not give you the best results. Time? Meh. The most basic tasks that don't have what you might call CPU-scale time requirements can be handled with the usual language and...

read more