Although I’m a programmer and not a sysadmin, I do know a bit about the latter. Today, a client asked me to install a new SSL certificate for their website which we’re still hosting.

Sure, how hard can it be?

Thing is, we don’t really do hosting anymore so installing SSL certificates is something I’m asked to do about once a year. Because of that, I’m hardly an expert and I’m often required to do a quick Google on stuff like the exact syntax of things.

In this case though, we got the new certificate from the soon-to-be new provider for this client, so it should be just a question of a quick SSH and an /etc/init.d/apache reload, amirite?

Well, it ended up taking me over an hour and a fair amount of head-desking. The server was refusing to reload/start. The error log wasn’t particularly helpful:

[Mon Jun 15 15:59:51.844898 2015] [ssl:emerg] [pid 576] AH02561: Failed to configure certificate www.xxx.com:443:0, check /etc/ssl/private/www.xxx.com.crt
[Mon Jun 15 15:59:51.844980 2015] [ssl:emerg] [pid 576] SSL Library Error: error:0906D06C:PEM routines:PEM_read_bio:no start line (Expecting: CERTIFICATE) -- Bad file contents or format - or even just a forgotten SSLCertificateKeyFile?
[Mon Jun 15 15:59:51.845000 2015] [ssl:emerg] [pid 576] SSL Library Error: error:140AD009:SSL routines:SSL_CTX_use_certificate_file:PEM lib
AH00016: Configuration Failed

Hm, so a borked certificate maybe? Seemed unlikely, but that was what Googlestackoverflow was suggesting when I searched the error. It seemed to be confirmed when manually validating the certificate file:

root@(none):/etc/ssl/private# openssl x509 -hash -noout -in www.xxx.com.crt
unable to load certificate
140591264368272:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE

However, file looked good to me – Unix line endings, definitely in the right format etc. I also compared it to their previous, now invalid certificate – at first glance, they looked alike (apart obviously from the actual certificate).

Then, just as I was about to give up and put back the old certificate for now, I saw it: for reasons unknown, the .crt began with ----BEGIN CERTIFICATE----- instead of -----BEGIN CERTIFICATE-----. That’s right, one freakin’ dash just cost me an hour of my life. I checked: the error was in the file I’d gotten from the hosting provider. Ouch… but I suppose an easy enough mistake to make when quickly copy/pasting stuff.

So, for anyone else losing their minds over this: the tags are delimited by 5 dashes (and the number of thy counting shall be 5). Not an entirely obvious mistake to spot, so a good one to check. 🙂