What The x.509 Is Wrong With Gmail?

Elephants Family

Image by Benh LIEU SONG

You don’t often stop to think about x.509 and the Public Key Infrastructure
(PKI) that authenticates our Internet connections. Allow me to explain why you

Transport Layer Security (TLS) uses x.509 certificates to authenticate
connections. In your every-day use of the Internet, this means that you get a
certificate from a server when you connect over HTTPS (for example.) This
certificate is the only reasonable means you have to verify the identity of a

Why does this matter? I’m glad you asked.

Just because a web site looks like your bank’s web site does not mean that it
is. The same is true of any server. Plenty of people would love to have you
enter your Gmail password, credit card information, or social security number
on a web site that is not what you think. In this past this was called wire
fraud. Now it is called identity theft.

There is one other reason that has become public recently. At least one agency
of the United States government has been said to perform man-in-the-middle
attacks against their targets. This means that they intercept connections.
They do this by responding to a web browser’s request faster than the website
from which the require was made. When the request uses HTTPS, the attacker
must also present a valid certificate.

I explained that we get a certificate when we connect to a server using HTTP.
However, in order to trust this certificate in the traditional sense, you must
trust the organization that signed the certificate (called the certificate
authority or CA.)

In my installation of Firefox 25.0.1, there are more than 90 trusted
certificate authorities. Do I really trust this huge list of organizations to
validate the identity of every site I visit? The answer varies from person to
person and organization to organization. I might trust GTE Corporation more
than I trust GoDaddy.com, Inc. yet, my browser is configured to trust each one
completely. (Some of these certificate authorities have also been accused of
signing fake certificates on behalf of government agencies.)

Is There a Solution?

From a security perspective, it would ideal if your bank gave you a copy of
their certificate when you went to the bank in person. Then you would have no
need to trust a certificate authority. (This is true of any public key
authentication scheme.) But this doesn’t solve the problem for Gmail or eBay.

Google has tried to provide a partial solution for Google Chrome users by
pinning certificates. (OWASP has a wiki page on the topic.) This is
nice for Google Chrome users, but doesn’t help if someone attacks your Android
phone’s connection to an IMAPS server.

The Perspectives project was started by Dan Wendlandt at CMU. The project is
an attempt to use the the concept of notary servers scattered around the
Internet to detect man-in-the-middle attacks. It has an easy-to-use Firefox
extension and provides a Python-based notary server so anyone can run a notary.

The TACK project by Moxie Marlinspike is Moxie’s most recent attempt to fix
our current PKI. (His previous attempt—the now defunct convergence.io—was
similar to the Perspectives Project.) TACK stands for Trust Assertions for
Certificate Keys. The project draft “defines a TLS Extension that enables a
TLS server to support ‘pinning’ to a self-chosen signing key.” In English, the
project adds a layer of indirection between the CA and the server’s certificate
to protect against rogue or compromised CAs.

Finally, several organizations, including EFF (SSL Observatory) and GRC
(Fingerprints) have projects which collect information about certificates
being served by various sites.

What about Gmail?

I’ve been a Gmail user for more than five years. Only recently have I decided
that I needed to evaluate the security trade-offs of using Gmail (and several
other web services.) As part of that, I started using the Perspectives
Project’s Firefox plugin and notary server. When I started to observe the
certificates coming back from Gmail, I noticed that the certificate my browser
receives never matches the certificates observed by Perspective notaries—even
the notary server running on the same system as my browser.

In this video, you can see that something very unusual happens when I connect
to mail.google.com with a web browser as opposed to using OpenSSL. (This
is true for several other Google services like sites.google.com.) Instead
of getting the same certificate (as identified by the certificate’s
fingerprint,) the two connections get different certificates.

You can try it for yourself from a shell, provided you have OpenSSL installed.
Use this command line. Don’t forget to press CTRL+D to close the connection:

$ openssl s_client -connect mail.google.com:443 -showcerts | openssl x509 -fingerprint -md5

I have not observed this behavior on any other sites or services on the
Internet. Initially, I thought someone was attacking my Gmail connection. I
have since observed the same behavior whether I connect from my home, my
Verizon wireless hotspot, or free public WiFi in an airport. I’ve also checked
with another tech-savvy Gmail user who lives in a different city and reports
the same behavior.

This leaves two possible causes. The first and more likely cause is that some
of Google’s servers send back different certificates depending on the TLS
request sent by the browser or OpenSSL command line tool. Alternately, some
party between Google and my laptop intercepts the connections and provides my
browser with a different but valid certificate. It seems far more likely that
Google is returning different certificates, specifically in pursuit of forward

In fact, Google wrote about their work two years ago. In my shock and
excitement, I had failed to check any Google sites in Safari or another browser
that doesn’t support forward secrecy (in conjunction with the ciphers accepted
by Google.) If I had, I would have noticed that other browsers get the same
certificate as the OpenSSL command line tool.

Have you seen this behavior on other sites? Did I reach the wrong conclusion?
Let me know in the comments, on Twitter or send me e-mail (tfarrell at this
domain; PGP key D79453F8).

Troy Farrell

Troy Farrell

Troy Farrell

Latest posts by Troy Farrell (see all)


Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.


  1. John

    I like the idea of trying to verify certificates against a database you actually trust in addition to the CA, but I think it could also cause problems with things like Zscaler (“Zscaler implements what is known as a man-in-the-middle attack…” – Wikipedia). I wonder how it would interact with those public WiFi redirects to the terms of service.

    • Troy Farrell

      I would expect an alternate mechanism of authenticating hosts and certificates would “cause problems” with things like Zscaler. That seems like a desirable problem. I realize that some enterprise environments may mandate such benevolent attackers, but it’s with the full knowledge that they are subverting the PKI. If you’re going to use something like Zscaler, you probably already have an enterprise CA installed on your client. Once that is in place, it seems like there would be little benefit to using something like Perspectives, except on the other side of the Zscaler proxy.

      I can tell you from experience that most public WiFi hotspots either require you to connect via HTTP before redirecting you to a secured connection (to avoid the browser’s warning page) or embrace the warning page and respond to your HTTPS request with a certificate that does not match the host name requested. In the case of public WiFi, I don’t have a good solution. I suspect that this is the very issue that causes Apple software to open a special browser that doesn’t scream loudly about an invalid certificate.