Tens of thousands of Chromebooks fail because of Symantec BlueCoat problem

Well, this is annoying. Maryland’s Montgomery County schools are using Chromebooks. The school system is using about 120,000 Chromebooks and multiple PCs running the Chrome web browser. But when Google recently updated them to Chrome OS 56, over 30 percent couldn’t log on, while many PCs running Chrome were unable to reach the web.

So, was it Google’s fault? Not so fast.

The school system was using Symantec’s BlueCoat, a man-in-the-middle (MitM) SSL web proxy. This uses ProxySG technology to examine Secure-Socket Layer (SSL) and Transport Layer Security (TLS) encrypted web content. So far, so good — if you want to make sure your seventh graders aren’t peeking into pornography. But, in this case, it turns our BlueCoat doesn’t support the newest standard web security protocol, TLS 1.3.

Special Feature

Cloud – How to Do SaaS Right

Software as a Service offers irresistible benefits for organizations of all sizes — from cost savings to scalability to mobile accessibility.

Whoops.

TLS is SSL’s successor. The newest version, TLS 1.3, blocks attacks that were effective against TLS 1.2 and earlier security protocols. It also speeds up web connections.

“This update, the first since 2008, is a major overhaul that provides both increased security and enhanced speed, especially on mobile networks,” said Nick Sullivan, CloudFlare‘s head of cryptography. “TLS 1.3 improves request speeds by requiring one less round trip to connect to an internet application, compared to previous versions, and can decrease page load times by 20 percent.”

Mozilla Firefox, Google Chrome, and Opera currently support TLS 1.3. Microsoft and Apple are working on supporting it in Internet Explorer 11 and Edge and Safari, respectively. Google decided in its latest update to more fully support only TLS 1.3.

Both Google and Mozilla saw TLS decryption problems coming. A recent security study found vendors are badly handling TLS inspections. The anti-virus or network appliances “terminate and decrypt the client-initiated TLS session, analyze the inner HTTP plaintext, and then initiate a new TLS connection to the destination website.” However, they then incorrectly validate certificates and can introduce security flaws.

Unfortunately, some programs, and BlueCoat’s is one of them, goes even farther wrong. They hiccup when trying to deal with TLS 1.3. What should happen is “Successful connection. Client and proxy may negotiate down to TLS 1.2 instead of TLS 1.3.” Instead, “when Chrome attempts to connect via TLS 1.3, BlueCoat hangs up connection.”

There are fixes. For example, you can force Chrome 56 to use TLS 1.2 with the flag:

chrome://flags/#ssl-version-max

at the next screen, change the flag from “Default” to “TLS 1.2”

But this has to be done by hand and it only works for the current user. It’s in no way a fix you’d want to use with tens of thousands of Chromebooks or PCs.

So, in the short-run, Google has set “Chrome so that when it can check-in will receive instructions to disable TLS 1.3 and thus should stay ‘fixed’ (for now).” To do this, you must set your web proxy so that it doesn’t intercept TLS traffic until all the devices have been upgraded. In the case of Chromebooks, that will simply be logging in. With PCs running Chrome, you’ll need to go to a Google site, such as Gmail, that requires a login. If you’re only installing Chrome OS or Chrome 56 now, the new versions default to using TLS 1.2 and should work fine.

Who’s fault it is?

Google puts the blame squarely on BlueCoat and other web proxy vendors. One note on the Chromium bug list said: “We’re waiting on a response from BlueCoat. They were made aware of TLS 1.3 several months ago, but evidently did not test their software per our instructions.”

Another Google software engineer stated: “These issues are always bugs in the middlebox products. TLS version negotiation is backwards compatible, so a correctly-implemented TLS-terminating proxy should not require changes to work in a TLS-1.3-capable ecosystem. It can simply speak TLS 1.2 at both client <-> proxy and proxy <-> server TLS connections. That these products broke is an indication of defects in their TLS implementations.”

Eventually, Google will return TLS 1.3 as the default. With Chrome 57 almost ready to go, this probably won’t happen until the Chrome 58 release, after the security vendors fix their proxies.

As for BlueCoat? A Symantec spokesperson said “Symantec has been alerted of a potential issue with TLS 1.3 on select devices. We’re investigating now and are working to resolve the issue.”

Hopefully it won’t take long. Otherwise, with Chromebooks being so popular in schools and SSL/TLS decryption being a common feature in educational-system web proxies, we can only expect to see further failures.

Related Stories:

VIDEO: Samsung’s new Chromebook Pro hybrid can run a universe of Android apps