Caddy 0.7.6 released

Hmm… I’m on my home network. I’ll try on my school network tomorrow and see if it works.

Not even sure where I would start debugging this. I tried using Google’s developer tools, but there’s nothing in there that seemed useful. I guess Wireshark is the next best thing. If anyone else has any suggestions I’m open to them!

Do you use a proxy of any sort at home? Or special router software or any monitoring software or anything of the sort? (Sorry to pry, just trying to figure out if this is a proxy-related issue.)

No proxy software and no monitoring software outside of antivirus, but I would think that would stop Firefox and IE from working too which isn’t the case. I tried out https://caddyserver.com my Mac using Chrome and it works fine. So my only way to reproduce this is with Google Chrome on my Windows 8 machine. Everything else works as normal.

This is bugging me so I did a little digging. By default, Caddy enables these 10 cipher suites (they come with the Go std lib):

  • TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  • TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  • TLS_RSA_WITH_AES_256_CBC_SHA
  • TLS_RSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
  • TLS_RSA_WITH_3DES_EDE_CBC_SHA

But over HTTPS, Caddy serves HTTP/2 by default. HTTP/2 blacklists almost 300 cipher suites for security reasons. 8 of the above cipher suites are on the list:

  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  • TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  • TLS_RSA_WITH_AES_256_CBC_SHA
  • TLS_RSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
  • TLS_RSA_WITH_3DES_EDE_CBC_SHA

So only two suites remain, except for the bold one which somehow slips through according to SSL Labs:

  • TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) ECDH 256 bits (eq. 3072 bits RSA) FS 128
  • TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a) ECDH 256 bits (eq. 3072 bits RSA) FS 256
  • TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009) ECDH 256 bits (eq. 3072 bits RSA) FS 128

(I do not think that third one showing up here is cause for alarm, although I’m not sure why it’s there at this point.)

I strongly suspect that if a connection fails to the Caddy site over HTTPS, it is because some relevant software only supports insecure cipher suites. This is unfortunate, but am not sure how to fix right now without compromising the security and stability of HTTP/2 or your websites.

If it wasn’t HTTP/2 you could just capture the traffic with Wireshark and compare. We need better HTTP/2 tooling…

True, http2 tooling is still scant, but do you think this error occurs at the TLS layer or the HTTP layer? If it occurs at the transport layer then maybe Wireshark could still be useful.

You’re right I do think it’s the transport layer and If you compare 2 captures (from a location that works, and a location that doesn’t) and everything looks normal then you can at-least eliminate the whole layer.

@alexmullins @netixen I’ve confirmed with Wireshark that my client is in fact sending 21 cipher suites to the server in the Client Hello:

The server responds with a FIN packet and outputs this to the log:

2015/10/12 18:30:51 http: TLS handshake error from 128.187.97.18:39703: tls: no cipher suite supported by both client and server
2015/10/12 18:30:51 http: TLS handshake error from 128.187.97.18:39713: tls: no cipher suite supported by both client and server
2015/10/12 18:30:51 http: TLS handshake error from 128.187.97.18:39728: tls: no cipher suite supported by both client and server
2015/10/12 18:30:51 http: TLS handshake error from 128.187.97.18:39746: tls: no cipher suite supported by both client and server

This reeks of software meddling in between. The client definitely accepts the cipher suites the server requires (at least in my case).

Ok, I’ll check Wireshark too and see what the results are on my end. @matt , this problem happens to you with any client while on your school network, correct? On my end I could only replicate this problem on my Windows 8.1 desktop running Google Chrome’s latest release (45.0.2454.101m); both Firefox and IE work. From your SS above I see that your client sent TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) and Caddy (Go stdlib) has that as one of its supported ciphers, so you’d expect the TLS connection to succeed.

When I connected using Chrome it sent 14 cipher suites and had TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) as the top one, so it shouldn’t have had a problem connecting using that cipher suite.

The protocol progression was:

TLSv1.2 - Client Hello
TLSv1.2 - Server Hello
TLSv1.2 - Certificate
TLSv1.2 - Server Key Exchange
TLSv1.2 - Server Hello Done
TLSv1.2 - Alert (Level: Fatal, Description: Decode Error)
TLSv1.2 - Alert (Level: Fatal,Description:Decode Error) Client Hello, Alert (Level: Fatal, Description: Unexpected Message)
TLSv1.1 - Client Hello
TLSv1.1 - Encrypted Alert

You can see my client is responding with fatal alerts (Decode Error and Unexpected Message). Also, it looks like Chrome is trying to reconnect using TLSv1.1, but that will not work with http2 as TLS1.2+ is required.

EDIT: It’s weird though, b/c for me this ONLY happens using Chrome. I’m starting to think that we are experiencing two separate issues. I could be wrong though.

1 Like

Wow, thanks for digging into this! That is a strange sequence of packets. Why would it have a decode error and downgrade to TLS 1.1? I’m using the exact same version of Chrome right now and it’s working fine. But I’m on OS X 10.10…

I’m beginning to believe with you that these are separate issues. I’m ~95% certain that there’s a proxy on my school’s campus running old software that is incompatible with the modern requirements. But I can’t debug it without access to it (right?). In your case it’s just your Chrome browser that has issues. Very odd.

I wonder if it’s an obscure bug specific to Chrome on Windows 8.1, maybe something with an underlying net library or a C binding that Chrome on Windows is using??

When I use the same version of Chrome on my Mac, I get these results in Wireshark – the highlighted line is where it appears yours encountered a decoding error:

So if these two experiments are analogous, we could assume it failed at the Client Key Exchange et. al. where it does most of the handshake work.

1 Like

The proxy is a blackbox but you can still reverse engineer what it’s doing by inspecting the front and back of the box :-P.

Try inspecting wireshark data from your computer and from the server, for the same problematic connection attempt. Look for suspicious differences.

Yea I have the same version on my Mac too and it works fine. I’ll keep looking into it later this week and report back. Gotta study for midterms now :cry:

Sounds good – I’m in the same boat :slight_smile: Also, Analytics tells me that ~12% of Windows visitors to the site are on 8.1, and 63% of Windows visitors are on Chrome. I just had a couple people on Twitter with Win8.1 verify that they were able to access it as well on Chrome. So… I’m stumped for now. :person_frowning:

@nictuku Good ideas, that’ll take a lot of time. But it may come to that.

Ahaahh! I have figured out what was going wrong on my end. After a few more Google searches and looking on the Chrome bug tracker I narrowed the problem down to my antivirus software (ofc!). I use Avast antivirus on my Windows 8.1 machine and it has a web shield that basically man-in-the-middles any https connection to check if the content contains malware (you can read more: http://security.stackexchange.com/a/73481). So I disabled that feature, connected to https://caddyserver.com and it worked! And thinking back, I ran into another problem with Avast preventing any Go programs I compiled from running. I had to turn off a ‘feature’ called DeepScan which scans unknown running files and stops anything ‘suspicious’. Now, I’d like to know why Avast is preventing the https connection to caddyserver.com and not the http one… More rabbit holes to chase!

2 Likes

Great find! That makes sense. I wonder if Avast suffers from similar neglect as my university’s campus proxy.

Also, I just saw this today: Chrome Canary has a HTTPS debugging tool. I’m excited to try this. (Edit: Just tried it. No help whatsoever. Oh well.)

Thanks for your help!

1 Like

Or… stop using antivirus. They are a scam, essentially.

1 Like

Nick, when I read that Avast is basically doing a MitM attack to sniff out malware in https connection it did bring up some questions in my mind. But not having an antivirus? Is that advisable? I’ve always had an antivirus program running on my Windows machines and keep it updated all the time. I’m not disagreeing with you, but just that I’ve heard the opposite, at least in regards to Windows.

Also, this CL is still in review: https://go-review.googlesource.com/#/c/15860/

2 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.