This is a merry tale of how I finally got back git over HTTP. And just in time
for the end of the year, how marvelous!
It’s almost spring, time to clean up the backlog! I wanted to publish this before the Christmas break but studying got a little out of hand.
I hope this information will be actually useful to others as from my research I should be the first one to figure out the problem and write about it.
For some time I have been self hosting software on a VPS, usually things are happy and I just keep the pace with upgrade but this problem sat with me for the better part of last year. For the first time I had broken infrastructure I did not know how to fix.
It might look like I know a lot of knobs and switches to make investigating the problem easier. I don’t. This is just me throwing time at a problem; this debugging mistery alone spanned more than 3 months and involved turning over a lot of rocks and no progress. Taking notes was the only way to go and it helped immensely. Here you will find the steps I went through and what kind of information they yielded.
At the same time this is a heavily edited version of the whole story and I gliss over a lot of logs reading, man pages, seaches, giving up, coming back and soul searching. It’s for your own good!
For some time I have been running an instance of gitea behind nginx as a reverse proxy. Everything was all-right until one day I could not make contact with it over git+HTTP.
I could not push or pull my remotes, the error persisted with different repositories and I was extremely confused as “it was working before”.
In October I finally resolved to put some time into this problem and gather more information. I opened an issue in my infrastructure repo and started writing what I knew.
Searching for others experiencing my problem yielded some results but no clear resolution steps or root cause. Maybe I was onto something new?
how to debug git
I needed to understand how to debug git and this behaviour without plunging into the deep portal that is the source code.
Another issue was that I had never debugged networking code and
therefore my chances of
survival success were very few. This is
definitely something that I ought to learn, debugging client-server
applications, it’s in the list, in a thousand years I’ll get to it.
Instead of sprinkling
printf around the source code of something
that I didn’t know I went and read the manual and found that you can
set some environment variables to make git a lot more willing to
share its secrets.
This is not too arcane knowledge but searching for the combination of “git debug” or “debug git” yield a lot of material about how to debug your code with git, not what we are interested in.
So anyway I was finally ready to unheart the misteries of my problem.
The next layer is actually figuring out what exactly is GnuTLS doing and whether I can bypass it.
To rule out my HTTP proxy configuration and SSL termination we better try and debug the HTTP connection before, just to make sure.
Uh, interesting. Let’s try GnuTLS.
What!? Both implementations have no problem performing an handshake, is git modifying some options when using libcurl as a transport?
And for a wonderfully confusing combination here is GnuTLS trying to be helpful.
Ok so we have gathered enough information, it’s not git, it’s not curl,
GnuTLS is very confused (beating yours truly by a wide margin) and OpenSSL
is happy. Sounds like the next step is to use the combo
to get us out of trouble.
Finally something easy, and I already have libcurl installed with the openssl flavour (yummy)! So now we have to specify the TLS backend to curl and everything will be ok!
Uh, what?! I asked nicely! Maybe I can get git to ask nicely
So it’s me and you GnuTLS, till death makes us apart.
Once my two neurons clicked together I remembered that I was also using a second instance of gitea, run by my partner in crime.
After a short email not only I got him to test his git client against my instance but I also got his apache configuration to compare to the one I was running with nginx (unfortunately no revelations there).
The results were very interesting
What kind of magic is this? He’s using OpenSSL and not GnuTLS! It’s not there in the logs but he confirmed me that he was not using GnuTLS, more intriguing.
Left without any hint at what was currently going on I ignored the problem a little more but not so long ago a new version of GnuTLS was released and I set to retry.
Woooo! Something changed, before we could not even get it to acknowledge we were running a server with TLS support.
Unfortunately the changelog for the package was not giving us any love.
But wait a minute! Does it mean that now it works? A little fiddling later it was clear that no, it would not work.
I was left with a client that could talk to all the servers except for mine and this left me with nothing more than to dump some more logs and read more carefully.
In the end there was another environment variable that I had still to try and it gave me some hints.
I had actually found about debugging GnuTLS before but this was the moment
it actually clicked. The important part from the previous log is
sent server name.
If you go back to the newer logs from
gnutls-cli-debug you can see that
it failed a test for
HTTPS server name.
I already knew about SNI but never bothered to actually set up my webserver correctly because it was “just working” when I deployed HTTPS to each one of my services. Now I suppose that most likely there was some session stored in memory that would allow nginx to make git go through but as time went on the session expired leaving me extremely confused.
There are many lessons to be learned from my story, make your tools debuggable with either configuration or environment variables, test important (breaking?) changes at different times, from different machines, from different implementations, take notes, ask for help and take your time to go through minutiae and ask yourself questions. This is the way1.
and yet I don’t have a shiny armour of beskar steel, how disappointing ↩