Let's Encrypt Certificate Installation Error: ACME Challenge Failures and Port 80 Conflicts

Introduction

Let's Encrypt has made SSL certificates free and accessible, which is fantastic. But the ACME protocol validation process has specific requirements that can trip you up, especially on servers with complex networking configurations. I recently set up a new server for a side project and ran into a frustrating Let's Encrypt installation error that took me several hours to resolve.

The error was not particularly descriptive — Certbot kept failing at the ACME HTTP-01 challenge with a timeout. The certificate authority could not reach my server on port 80 to verify domain ownership. Since I had successfully installed Let's Encrypt certificates on dozens of servers before, I was confused about why this particular setup was different.

Environment

Server: Hetzner Cloud CX22, Ubuntu 22.04
Web server: Nginx 1.24.0
Domain: myapp.example.com
Certbot: 2.1.0 (snap)
Firewall: UFW enabled with ports 22, 80, 443 open
Cloud provider firewall: Hetzner Cloud Firewall with port 80 and 443 open

The server was freshly provisioned with Nginx installed and serving a basic welcome page on both HTTP and HTTPS (self-signed cert).

Problem

When I ran Certbot, I got the following error:

sudo certbot --nginx -d myapp.example.com

# Output:
# Saving debug log to /var/log/letsencrypt/letsencrypt.log
# Requesting a certificate for myapp.example.com
#
# Performing the following challenges:
# http-01 challenge for myapp.example.com
# Waiting for verification...
# Challenge failed for domain myapp.example.com
# http-01 challenge for myapp.example.com
# Cleaning up challenges
# Some challenges have failed.

The debug log was more informative:

2025-02-15 10:23:45,678:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/snap/certbot/3456/lib/python3.10/site-packages/certbot/_internal/auth_handler.py", line 98, in handle_authorizations
    self._challenge_validator.authorize(log.debug)
  File "/snap/certbot/3456/lib/python3.10/site-packages/certbot/_internal/challenge_costs.py", line 87, in _check_authorization
    raise errors.AuthorizationError(f"Challenge failed: {challenge}")
certbot._internal.errors.AuthorizationError: Challenge failed: http-01

2025-02-15 10:23:45,679:WARNING:certbot._internal.auth_handler:Http challenge failed:
certbot._internal.errors.AuthorizationError: Unable to set up temporary challenge server: [Errno 98] Address already in use

The key line was Address already in use — Certbot was trying to start a temporary server on port 80 to handle the ACME challenge, but something else was already listening on that port.

Analysis

I checked what was using port 80:

sudo ss -tlnp | grep :80
# LISTEN  0  511  0.0.0.0:80  0.0.0.0:*  users:(("nginx",pid=1234,fd=6))

Nginx was already listening on port 80. This was expected — Nginx was serving the welcome page. The issue was that when Certbot runs in standalone mode (the default), it tries to start its own temporary web server on port 80 to handle the ACME challenge. Since Nginx was already on that port, it failed.

The solution should have been simple: use the Nginx plugin instead of standalone mode. But I had tried that and it failed too:

sudo certbot --nginx -d myapp.example.com
# Same error as above

The Nginx plugin should have injected a location block into Nginx to handle the challenge without needing a separate server. The fact that it still failed meant something else was wrong.

I enabled Certbot debug mode and checked the Nginx configuration:

sudo certbot --nginx -d myapp.example.com --debug

The debug output revealed that Certbot was successfully creating the challenge file in the webroot, but the ACME server could not reach it. I verified by manually creating a test file:

sudo mkdir -p /var/www/certbot/.well-known/acme-challenge/
echo "test" | sudo tee /var/www/certbot/.well-known/acme-challenge/test-file
curl http://myapp.example.com/.well-known/acme-challenge/test-file
# Expected: "test"
# Actual: 404 Not Found

The 404 response confirmed that Nginx was not configured to serve files from the Certbot webroot. The Nginx configuration did not have a location block for /.well-known/acme-challenge/.

Solution

I updated the Nginx configuration to properly handle ACME challenges. I created a dedicated configuration snippet at /etc/nginx/snippets/acme-challenge.conf:

# ACME challenge handling for Let's Encrypt
location /.well-known/acme-challenge/ {
    root /var/www/certbot;
    allow all;
}

# Redirect all other HTTP traffic to HTTPS
location / {
    return 301 https://$host$request_uri;
}

Then I included this in the server block:

server {
    listen 80;
    server_name myapp.example.com;

    include /etc/nginx/snippets/acme-challenge.conf;
}

After reloading Nginx:

sudo nginx -t && sudo systemctl reload nginx
# nginx: configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

I verified the challenge file was now accessible:

curl http://myapp.example.com/.well-known/acme-challenge/test-file
# test

Now Certbot worked:

sudo certbot --nginx -d myapp.example.com

# Output:
# Requesting a certificate for myapp.example.com
# Successfully received certificate.
# Certificate is saved at: /etc/letsencrypt/live/myapp.example.com/fullchain.pem
# Key is saved at:       /etc/letsencrypt/live/myapp.example.com/privkey.pem
# This certificate expires on 2025-05-16.
# Your account credentials have been saved in the Certbot configuration file.

For ongoing maintenance, I added a cron job to keep the Certbot webroot clean and verify challenge file accessibility:

#!/bin/bash
# /opt/scripts/verify-acme-setup.sh

WEBROOT="/var/www/certbot"
TEST_FILE="$WEBROOT/.well-known/acme-challenge/health-check"

mkdir -p "$(dirname "$TEST_FILE")"
echo "healthy" > "$TEST_FILE"

# Verify from localhost
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/.well-known/acme-challenge/health-check)
if [ "$HTTP_CODE" != "200" ]; then
    echo "ALERT: ACME challenge webroot not accessible (HTTP $HTTP_CODE)"
fi

rm -f "$TEST_FILE"

Lessons Learned

Always include the ACME challenge location — If you are running a web server, it must have a location block for /.well-known/acme-challenge/ that serves from the Certbot webroot.
Check both layers of firewalls — Cloud providers often have their own firewall on top of the OS firewall. Make sure port 80 is open in both.
Use the Nginx plugin when possible — certbot --nginx is simpler than standalone mode because it configures Nginx automatically. But you still need the webroot to be accessible.
Test with curl — Always verify that challenge files are accessible from the public internet before running Certbot. This saves time on debugging.
Keep the webroot writable — Ensure the Certbot process has write permissions to the webroot directory. Permission issues are a common cause of silent failures.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.