tooolbox/blog/Git LFS Speed Issues on Linux: Diagnosis and Optimization
performance2024-07-10·12 min·288/348

Git LFS Speed Issues on Linux: Diagnosis and Optimization

Resolving slow Git LFS operations on Linux, including bandwidth optimization, parallel downloads, and troubleshooting transfer bottlenecks.

Introduction

Git LFS (Large File Storage) is essential for managing large binary files in Git repositories, but it can be painfully slow without proper configuration. Slow clone times, hanging transfers, and bandwidth limitations are common complaints. This post covers diagnosing and optimizing Git LFS performance on Linux.

Environment

The examples use Git 2.39.2 with Git LFS 3.4.0 on Ubuntu 22.04 LTS, working with a repository containing 50 GB of binary assets.

git --version
# git version 2.39.2

git lfs --version
# git-lfs/3.4.0 (GitHub; linux amd64; go 1.21.5)

Problem

Cloning a repository with Git LFS is extremely slow:

git clone https://github.com/example/large-repo.git
# Cloning into 'large-repo'...
# Receiving objects: 100% (12345/12345), done.
#Downloading LFS objects:  12% (1234/10000), 1.2 GB/10 GB, 00:45:12 ETA

# Transfer stalls frequently
# git lfs pull takes hours for modest changes

Analysis

Git LFS performance issues are caused by:

  1. Single-threaded transfers: Default config limits parallelism
  2. Small batch sizes: Frequent HTTP requests add overhead
  3. Network latency: Each LFS object requires separate HTTP request
  4. Server-side throttling: GitHub/GitLab may rate-limit
  5. Local disk I/O: Writing large files sequentially

Check current LFS configuration:

git lfs env
# git-lfs/3.4.0
#     ENDPOINT: https://github.com/example/large-repo.git/info/lfs
#     LOCALREF: refs/heads/main
#     CONCURRENCY: 8
#     TUSTRANSFERS: true
#     SKIPSMALLOBJECTS: false

Solution

1. Increase concurrency

# Set concurrent transfers (default: 8)
git config --global lfs.concurrenttransfers 16

# For high-bandwidth connections, increase further
git config --global lfs.concurrenttransfers 32

2. Optimize batch size

# Increase the number of objects per batch request
git config --global lfs.batchsize 256

# Default is 100, larger values reduce HTTP overhead

3. Use Git LFS with parallel operations

# Enable parallel fetching
git config --global fetch.parallel 4

# Use multiple threads for checkout
git config --global core.preloadindex true
git config --global core.fscache true

4. Configure Git for large repositories

# Increase buffer size
git config --global http.postBuffer 524288000

# Increase low speed limit (bytes/sec)
git config --global http.lowSpeedLimit 1000

# Set timeout (seconds)
git config --global http.lowSpeedTime 60

5. Use a faster network transport

# For GitHub, use the SSH protocol
git remote set-url origin git@github.com:example/large-repo.git

# For GitLab, enable compression
git config --global http.compress true

6. Optimize LFS storage backend

# Use local LFS cache
git config --global lfs.storage /mnt/fast-ssd/lfs-cache

# Enable LFS object verification skip (faster but less safe)
git config --global lfs.skipinitialdownload false

7. Resume failed transfers

# LFS automatically retries failed transfers
# Check retry configuration
git lfs env

# Force a retry of failed objects
git lfs pull --include="*.psd"

# Resume a partial clone
git lfs fetch --all
git lfs checkout

8. Monitor transfer progress

# Verbose output for debugging
GIT_TRACE=1 GIT_LFS_TRACE=1 git lfs pull
# trace: built-in: git lfs pull
# trace: run: git-lfs pull
# trace: exec: git-lfs pull
# trace: try_git-lfs: git-lfs pull
# trace: cmd.Run: resolve C:\Program Files\Git\cmd\git-lfs.EXE

9. Use specific file patterns

# Only download specific file types
git lfs pull --include="*.zip,*.tar.gz"

# Exclude files you do not need locally
git lfs pull --exclude="*.bak,*.tmp"

10. Configure LFS for CI/CD

# In CI, use shallow clone with LFS
git clone --depth 1 https://github.com/example/large-repo.git
git lfs install
git lfs pull

# Or use a pre-built LFS cache
export CI_LFS_CACHE=/cache/lfs
git config --global lfs.storage $CI_LFS_CACHE

11. Check network performance

# Test download speed
curl -o /dev/null -w "speed: %{speed_download}\n" https://example.com/lfs/test

# Check MTU for large transfers
ping -M do -s 8972 github.com
# If this fails, your MTU may be too low for LFS

12. Use Git LFS with Docker volumes

# Mount a volume for LFS cache
docker run -v /mnt/lfs-cache:/root/.git/lfs \
  -v $(pwd):/repo \
  -w /repo \
  git-lfs/git-lfs pull

Lessons Learned

Git LFS performance is primarily limited by network bandwidth and HTTP overhead. Increasing concurrenttransfers and batchsize are the most impactful changes. For very large repositories, consider using a local LFS storage backend on fast SSD storage. And always monitor transfers with GIT_LFS_TRACE=1 to identify bottlenecks.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.