Downloading Large Files on Linux: wget and curl Mastery

Introduction

Downloading large files on Linux is a common task, whether pulling Docker images, fetching datasets, or retrieving software packages. wget and curl are the two primary tools for this, and each has strengths. This post covers efficient download techniques, resume support, and best practices for handling large file transfers.

Environment

The examples use wget 1.21.3 and curl 7.88.1 on Ubuntu 22.04 LTS. The files being downloaded range from 1 GB to 100 GB.

wget --version | head -1
# GNU Wget 1.21.3 built on linux-gnu.

curl --version | head -1
# curl 7.88.1 (x86_64-pc-linux-gnu)

Problem

Large file downloads frequently fail due to network interruptions, server timeouts, or connection drops. Without proper tools and techniques, a failed download means starting over from scratch.

wget https://example.com/large-dataset.tar.gz
# ... (download fails at 45%)
# ERROR: cannot verify example.com's certificate.
# Read error: Connection reset by peer.

Analysis

Key considerations for large file downloads:

1. Resume support: Ability to continue from where the download stopped
2. Checksum verification: Ensure downloaded file integrity
3. Parallel downloads: Split file into parts for faster transfer
4. Timeout handling: Handle slow or unresponsive servers
5. Authentication: Handle protected resources

Solution

1. Basic wget download with resume

# Continue a failed download
wget -c https://example.com/large-file.tar.gz

# The -c flag resumes from where it left off

2. wget with retry and timeout settings

wget --tries=10 \
     --wait=5 \
     --read-timeout=60 \
     --timeout=120 \
     --continue \
     https://example.com/large-file.tar.gz

3. wget with progress and logging

wget --progress=bar:force:noscroll \
     --server-response \
     -o download.log \
     https://example.com/large-file.tar.gz

4. curl with resume support

curl -C - -O https://example.com/large-file.tar.gz

# The -C - flag enables resume
# -O saves with remote filename

5. curl with retry and timeout

curl --retry 10 \
     --retry-delay 5 \
     --retry-max-time 600 \
     --connect-timeout 30 \
     --max-time 3600 \
     -L -O \
     https://example.com/large-file.tar.gz

6. Verify download integrity

# Download checksum file
wget https://example.com/large-file.tar.gz.sha256

# Verify
sha256sum -c large-file.tar.gz.sha256
# large-file.tar.gz: OK

# Or manually
sha256sum large-file.tar.gz
# abc123...  large-file.tar.gz

7. Parallel downloads with split

# Split download into multiple parts
for i in $(seq 1 8); do
  wget -c https://example.com/large-file.tar.gz.part${i} &
done
wait

# Combine parts
cat large-file.tar.gz.part* > large-file.tar.gz

8. Use aria2 for parallel downloads

# Install aria2
sudo apt-get install aria2

# Download with multiple connections
aria2c -x 16 -s 16 https://example.com/large-file.tar.gz
# -x 16: 16 connections per server
# -s 16: split into 16 segments

# Resume with aria2
aria2c -c https://example.com/large-file.tar.gz

9. Download behind authentication

# Basic auth with wget
wget --user=admin --password=secret https://example.com/protected/file.tar.gz

# Basic auth with curl
curl -u admin:secret -O https://example.com/protected/file.tar.gz

# Token-based auth
curl -H "Authorization: Bearer TOKEN" -O https://example.com/protected/file.tar.gz

10. Download through a proxy

# wget with proxy
wget -e http_proxy=http://proxy:8080 \
     -e https_proxy=http://proxy:8080 \
     https://example.com/large-file.tar.gz

# curl with proxy
curl -x http://proxy:8080 -O https://example.com/large-file.tar.gz

11. Monitor download progress

# wget: Use --progress
wget --progress=dot:mega https://example.com/large-file.tar.gz

# curl: Use -# for progress bar
curl -# -O https://example.com/large-file.tar.gz

# Monitor bandwidth
nload eth0

12. Download with rate limiting

# wget: Limit bandwidth to 1MB/s
wget --limit-rate=1m https://example.com/large-file.tar.gz

# curl: Limit bandwidth to 1MB/s
curl --limit-rate 1M -O https://example.com/large-file.tar.gz

13. Batch download with script

#!/bin/bash
# download.sh - Download multiple files with retry

URLS=(
  "https://example.com/file1.tar.gz"
  "https://example.com/file2.tar.gz"
  "https://example.com/file3.tar.gz"
)

for url in "${URLS[@]}"; do
  filename=$(basename "$url")
  wget -c --tries=5 --wait=10 "$url" -O "$filename"
  if [ $? -eq 0 ]; then
    sha256sum "$filename" >> checksums.txt
    echo "Downloaded: $filename"
  else
    echo "Failed: $filename"
  fi
done

Lessons Learned

For simple downloads, wget with -c (resume) and --tries (retry) is sufficient. For complex scenarios with parallel connections, aria2 is the superior choice. Always verify checksums after downloading large files. And for automated downloads, use scripts with retry logic and logging to handle transient failures gracefully.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.