Nginx Reverse Proxy Configuration Error: 502 Bad Gateway and Upstream Timing

Introduction

Nginx is probably the most common reverse proxy in the world, and I have used it extensively for routing traffic to my various applications. Last month, I deployed a new Go microservice behind Nginx and immediately started seeing 502 Bad Gateway errors. The strange part was that it only happened under moderate load — maybe 50 concurrent connections. At low traffic, everything worked perfectly fine.

This turned out to be one of those issues where the default Nginx configuration is good enough for most cases, but falls apart the moment you push it slightly. The root cause was a combination of upstream connection limits and proxy buffer sizing that I had never needed to tune before. Here is the full story.

Environment

• Nginx version: 1.24.0 (Ubuntu 22.04 package)
• Upstream application: Go 1.21 HTTP server on port 8080
• OS: Ubuntu 22.04 LTS on an AWS EC2 t3.medium instance
• Client traffic: Load test using wrk with 100 concurrent connections, 10-second duration
• Network: All services on the same VPC, communicating over localhost

The Nginx configuration was fairly standard — a server block listening on port 443 with SSL termination, proxying requests to the Go service on 127.0.0.1:8080.

Problem

The 502 errors appeared in the Nginx error log as follows:

2025/03/10 14:23:45 [error] 28413#28413: *1247 upstream prematurely closed connection while reading response header from upstream, client: 10.0.1.55, server: api.example.com, request: "POST /v1/orders HTTP/1.1", upstream: "http://127.0.0.1:8080/v1/orders", host: "api.example.com"
2025/03/10 14:23:45 [error] 28413#28413: *1247 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.1.55, server: api.example.com, upstream: "http://127.0.0.1:8080/v1/orders", host: "api.example.com"

The first error line indicates that the upstream Go service closed the connection before sending a response header. The second error indicates that Nginx could not even connect to the upstream at all. The combination of these two errors pointed to the Go service crashing or hitting some resource limit under load.

But when I ran the Go service standalone (without Nginx in front), it handled the same load without any issues. So the problem was specifically in the Nginx-to-upstream communication.

Analysis

I started by checking the Go service's connection handling. It was using Go's default http.Server settings, which includes no limit on concurrent connections. The Go service was healthy.

Next, I examined the Nginx configuration more carefully. The key section was the location block:

location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

This is a minimal configuration that relies entirely on Nginx defaults. The problem was that I had not configured proxy_buffering or proxy_connect_timeout properly. By default, Nginx uses:

• proxy_connect_timeout: 60 seconds (this was fine)
• proxy_read_timeout: 60 seconds (this was fine)
• proxy_buffers: 8 4k or 8k pages (this was the problem)
• worker_connections: 512 (default for many installations)

With 100 concurrent connections, each request being proxied consumed a proxy buffer. The default buffer configuration was too small for the response sizes my Go service was returning (JSON payloads averaging 50KB each). Nginx was running out of buffer space and dropping connections.

I confirmed this by checking the connection state:

ss -s
# Output showed:
# TCP:   612 (estab 508, closed 104, orphaned 12, timewait 104)
# The 508 established connections were close to the worker_connections limit

Solution

I updated the Nginx configuration with proper buffer and connection settings:

http {
    worker_processes auto;
    worker_connections 1024;

    upstream backend {
        server 127.0.0.1:8080;
        keepalive 32;
    }

    server {
        listen 443 ssl http2;
        server_name api.example.com;

        location / {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_buffering on;
            proxy_buffer_size 16k;
            proxy_buffers 8 32k;
            proxy_busy_buffers_size 64k;

            proxy_connect_timeout 10s;
            proxy_read_timeout 30s;
            proxy_send_timeout 30s;
        }
    }
}

The critical changes were:

1. Increased proxy_buffers — From the default 8 4k/8k to 8 32k, giving Nginx enough space to buffer upstream responses
2. Added proxy_buffer_size — Set to 16k to handle larger response headers
3. Enabled keepalive connections — The upstream block with keepalive 32 reduced the overhead of establishing new connections for each request
4. Set proxy_http_version 1.1 — Required for keepalive connections to work properly
5. Increased worker_connections — From 512 to 1024 to handle more concurrent connections

After reloading Nginx, I re-ran the load test:

wrk -t4 -c100 -d10s http://api.example.com/v1/orders
# Results:
#   Requests/sec: 1247.83
#   Transfer/sec: 145.22MB
#   0 errors in 10 seconds

Zero errors. The throughput also improved significantly because keepalive connections eliminated the TCP handshake overhead on every request.

Lessons Learned

1. Do not rely on Nginx defaults for production — The default buffer sizes are designed for small responses. If your upstream returns anything larger than a few kilobytes, you need to tune proxy_buffers.
2. Enable upstream keepalive — This is one of the most impactful changes you can make for reverse proxy performance. It reduces latency and connection overhead.
3. Check worker_connections — If you are proxying to a single upstream, your effective connection limit is worker_connections times worker_processes. Plan accordingly.
4. Always test under realistic load — The issue only appeared at 50+ concurrent connections. Development environments with single-user testing will never surface these problems.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.