The issue at hand was when running certbot to obtain an SSL certificate the following error would occur:
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.mydomain.com
Waiting for verification…
Challenge failed for domain www.mydomain.com
http-01 challenge for www.mydomain.com
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
– The following errors were reported by the server:
Type: unauthorized
Detail: During secondary validation:
Invalid response from http://www.domain.com/.well-known/acme-challenge
At first I thought there was a problem with writing the challenge to my HTTP site but checking the site the TXT record was there but Let’s Encrypt could not read it. I went through the usual suspects: checked that port 80 was open, that there was HTTP access to the site and even verified that the TXT response was visible from an alternate network, www.mxtoolbox.com. I was extremely frustrated that I could not fin the issue so I gave up and decided to use DNS to validate the domain.
I immediately became suspicious when Let’s Encrypt could not validate using DNS either! A light went on and I though about the possibility that the chanllenge response server might be outside the US so I temporarily disabled geo-fencing on our gateway and bingo! The challenge response HTTP-01 worked like a charm.
If you have racked your brain on this like me, try disabling geo-fencing or IP restrictions until the challenge-response completes. I tried to get a list of countries/IP address but they are not published for security reasons.