A tale of SNI and git
This is a merry tale of how I finally got back git over HTTP. And just in time
for the end of the year, how marvelous!
It’s almost spring, time to clean up the backlog! I wanted to publish this before the Christmas break but studying got a little out of hand.
I hope this information will be actually useful to others as from my research I should be the first one to figure out the problem and write about it.
For some time I have been self hosting software on a VPS, usually things are happy and I just keep the pace with upgrade but this problem sat with me for the better part of last year. For the first time I had broken infrastructure I did not know how to fix.
It might look like I know a lot of knobs and switches to make investigating the problem easier. I don’t. This is just me throwing time at a problem; this debugging mistery alone spanned more than 3 months and involved turning over a lot of rocks and no progress. Taking notes was the only way to go and it helped immensely. Here you will find the steps I went through and what kind of information they yielded.
At the same time this is a heavily edited version of the whole story and I gliss over a lot of logs reading, man pages, seaches, giving up, coming back and soul searching. It’s for your own good!
our problem
For some time I have been running an instance of gitea behind nginx as a reverse proxy. Everything was all-right until one day I could not make contact with it over git+HTTP.
$ git push
fatal: unable to access 'https://git.edoput.it/edoput/blog.git/': gnutls_handshake() failed: Error in protocol versionI could not push or pull my remotes, the error persisted with different repositories and I was extremely confused as “it was working before”.
In October I finally resolved to put some time into this problem and gather more information. I opened an issue in my infrastructure repo and started writing what I knew.
Searching for others experiencing my problem yielded some results but no clear resolution steps or root cause. Maybe I was onto something new?
how to debug git
I needed to understand how to debug git and this behaviour without plunging into the deep portal that is the source code.
Another issue was that I had never debugged networking code and
therefore my chances of survival success were very few. This is
definitely something that I ought to learn, debugging client-server
applications, it’s in the list, in a thousand years I’ll get to it.
Instead of sprinkling printf around the source code of something
that I didn’t know I went and read the manual and found that you can
set some environment variables to make git a lot more willing to
share its secrets.
export GIT_CURL_VERBOSE=1
export GIT_TRACE_PACKET=2This is not too arcane knowledge but searching for the combination of “git debug” or “debug git” yield a lot of material about how to debug your code with git, not what we are interested in.
So anyway I was finally ready to unheart the misteries of my problem.
* Couldn't find host git.edoput.it in the .netrc file; using defaults
* Trying 51.15.250.122:443...
* TCP_NODELAY set
* Connected to git.edoput.it (51.15.250.122) port 443 (#0)
* found 363 certificates in /etc/ssl/certs
* ALPN, offering h2
* ALPN, offering http/1.1
* gnutls_handshake() failed: Error in protocol version
* Closing connection 0
fatal: unable to access 'https://git.edoput.it/edoput/blog.git/': gnutls_handshake() failed: Error in protocol versionDamn it!
debugging GnuTLS
The next layer is actually figuring out what exactly is GnuTLS doing and whether I can bypass it.
To rule out my HTTP proxy configuration and SSL termination we better try and debug the HTTP connection before, just to make sure.
$ openssl s_client -connect git.edoput.it:443
CONNECTED(00000003)
...
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
SSL handshake has read 3106 bytes and written 385 bytes
Verification: OK
---
...
---
Post-Handshake New Session Ticket arrived:
...Uh, interesting. Let’s try GnuTLS.
$ gnutls-cli -V -p 443 git.edoput.it
Processed 120 CA certificate(s).
Resolving 'git.edoput.it:443'...
Connecting to '51.15.250.122:443'...
...
- Handshake was completed
- Simple Client Mode:What!? Both implementations have no problem performing an handshake, is git modifying some options when using libcurl as a transport?
And for a wonderfully confusing combination here is GnuTLS trying to be helpful.
$ gnutls-cli-debug -V -p 443 git.edoput.it
GnuTLS debug client 3.6.9
Checking git.edoput.it:443
whether the server accepts default record size (512 bytes)... no
whether %ALLOW_SMALL_RECORDS is required... no
whether we need to disable TLS 1.2... yes
whether we need to disable TLS 1.1... yes
whether we need to disable TLS 1.0... yes
whether %NO_EXTENSIONS is required... yes
whether %COMPAT is required... yes
for TLS 1.0 (RFC2246) support... no
for TLS 1.0 (RFC2246) support with TLS 1.0 record version... no
for TLS 1.1 (RFC4346) support... no
fallback from TLS 1.1 to... failed
for TLS 1.2 (RFC5246) support... no
Server does not support any of SSL 3.0, TLS 1.0, 1.1, 1.2 and 1.3Ok so we have gathered enough information, it’s not git, it’s not curl,
GnuTLS is very confused (beating yours truly by a wide margin) and OpenSSL
is happy. Sounds like the next step is to use the combo git + libcurl + openssl
to get us out of trouble.
Finally something easy, and I already have libcurl installed with the openssl flavour (yummy)! So now we have to specify the TLS backend to curl and everything will be ok!
$ CURL_SSL_BACKEND=OpenSSL git push # here trying both OpenSSL or openssl had the same outcome
fatal: unable to access 'https://git.edoput.it/edoput/blog.git/': gnutls_handshake() failed: Error in protocol versionUh, what?! I asked nicely! Maybe I can get git to ask nicely
$ git config http.sslBackend openssl
$ git push
fatal: Unsupported SSL backend 'openssl'. Supported SSL backends:
gnutlsSo it’s me and you GnuTLS, till death makes us apart.
external help
Once my two neurons clicked together I remembered that I was also using a second instance of gitea, run by my partner in crime.
After a short email not only I got him to test his git client against my instance but I also got his apache configuration to compare to the one I was running with nginx (unfortunately no revelations there).
The results were very interesting
slash$ git push
* Couldn't find host git.edoput.it in the .netrc file; using defaults
* Trying 51.15.250.122:443...
* TCP_NODELAY set
* Connected to git.edoput.it (51.15.250.122) port 443 (#0)
* ALPN, offering h2
...
<
* Connection #0 to host git.edoput.it left intact
10:50:32.045409 pkt-line.c:80 packet: git< # service=git-receive-pack
10:50:32.045448 pkt-line.c:80 packet: git< 0000
10:50:32.045465 pkt-line.c:80 packet: git< f034b9d2db83b2d4bd196bd89ba776141b159282 refs/heads/master\0report-status delete-refs side-band-64k quiet atomic ofs-delta agent=git/2.20.1
10:50:32.045485 pkt-line.c:80 packet: git< 0000
Everything up-to-dateWhat kind of magic is this? He’s using OpenSSL and not GnuTLS! It’s not there in the logs but he confirmed me that he was not using GnuTLS, more intriguing.
Left without any hint at what was currently going on I ignored the problem a little more but not so long ago a new version of GnuTLS was released and I set to retry.
GnuTLS debug client 3.6.10
Checking git.edoput.it:443
whether the server accepts default record size (512 bytes)... no
whether %ALLOW_SMALL_RECORDS is required... no
whether we need to disable TLS 1.2... yes
whether we need to disable TLS 1.1... yes
whether we need to disable TLS 1.0... yes
whether %NO_EXTENSIONS is required... yes
whether %COMPAT is required... yes
for TLS 1.0 (RFC2246) support... no
for TLS 1.0 (RFC2246) support with TLS 1.0 record version... no
for TLS 1.1 (RFC4346) support... no
fallback from TLS 1.1 to... failed
for TLS 1.2 (RFC5246) support... no
for TLS 1.3 (RFC8446) support... yes
for known TLS or SSL protocols support... yes
TLS1.2 neg fallback from TLS 1.6 to... failed (server requires fallback dance)
for HTTPS server name... failed
for certificate chain order... unsorted
for safe renegotiation (RFC5746) support... no
for encrypt-then-MAC (RFC7366) support... no
for ext master secret (RFC7627) support... no
for heartbeat (RFC6520) support... no
for version rollback bug in RSA PMS... yes
whether the server ignores the RSA PMS version... no
whether small records (512 bytes) are tolerated on handshake... no
whether cipher suites not in SSL 3.0 spec are accepted... no
whether a bogus TLS record version in the client hello is accepted... no
whether the server understands TLS closure alerts... no
whether the server supports session resumption... no
for anonymous authentication support... no
for RSA key exchange support... no
for ephemeral Diffie-Hellman support... no
for RFC7919 Diffie-Hellman support... no
for ephemeral EC Diffie-Hellman support... no
for curve SECP256r1 (RFC4492)... no
for curve SECP384r1 (RFC4492)... no
for curve SECP521r1 (RFC4492)... no
for curve X25519 (RFC8422)... no
for AES-GCM cipher (RFC5288) support... no
for AES-CCM cipher (RFC6655) support... no
for AES-CCM-8 cipher (RFC6655) support... no
for AES-CBC cipher (RFC3268) support... no
for CAMELLIA-GCM cipher (RFC6367) support... no
for CAMELLIA-CBC cipher (RFC5932) support... no
for 3DES-CBC cipher (RFC2246) support... no
for ARCFOUR 128 cipher (RFC2246) support... no
for CHACHA20-POLY1305 cipher (RFC7905) support... no
for MD5 MAC support... no
for SHA1 MAC support... no
for SHA256 MAC support... no
for max record size (RFC6066) support... no
for OCSP status response (RFC6066) support... noWoooo! Something changed, before we could not even get it to acknowledge we were running a server with TLS support.
Unfortunately the changelog for the package was not giving us any love.
But wait a minute! Does it mean that now it works? A little fiddling later it was clear that no, it would not work.
Conclusions
I was left with a client that could talk to all the servers except for mine and this left me with nothing more than to dump some more logs and read more carefully.
In the end there was another environment variable that I had still to try and it gave me some hints.
$ GNUTLS_DEBUG_LEVEL=2 git push
gnutls[2]: Enabled GnuTLS 3.6.9 logging...
gnutls[2]: getrandom random generator was detected
gnutls[2]: Intel SSSE3 was detected
...
gnutls[2]: HSK[0x56004e52a0a0]: sent server name: 'git.edoput.it'
* gnutls_handshake() failed: Error in protocol version
* Closing connection 0
fatal: unable to access 'https://git.edoput.it/edoput/blog.git/': gnutls_handshake() failed: Error in protocol versionI had actually found about debugging GnuTLS before but this was the moment
it actually clicked. The important part from the previous log is
sent server name.
If you go back to the newer logs from gnutls-cli-debug you can see that
it failed a test for HTTPS server name.
I already knew about SNI but never bothered to actually set up my webserver correctly because it was “just working” when I deployed HTTPS to each one of my services. Now I suppose that most likely there was some session stored in memory that would allow nginx to make git go through but as time went on the session expired leaving me extremely confused.
There are many lessons to be learned from my story, make your tools debuggable with either configuration or environment variables, test important (breaking?) changes at different times, from different machines, from different implementations, take notes, ask for help and take your time to go through minutiae and ask yourself questions. This is the way1.
-
and yet I don’t have a shiny armour of beskar steel, how disappointing ↩