Configuring Automatic Service Restarts with systemd

Services crash occasionally, but systemd can automatically restart them. Setting up proper restart policies ensures high availability without manual intervention. I configured auto-restart for a critical trading application that occasionally crashed during market volatility.

Environment

• OS: Ubuntu 22.04 LTS
• Service: Custom trading application
• Goal: Automatic restart on failure with backoff

$ systemctl --version
systemd 249 (249.11-0ubuntu3.12)

The Problem

# Service crashes occasionally
$ sudo systemctl status trading-app
● trading-app.service - Trading Application
     Loaded: loaded (/etc/systemd/system/trading-app.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2025-05-15 14:30:00 WET; 2min ago

# Manual restart required each time
$ sudo systemctl restart trading-app

Analysis

Check Current Service Configuration

$ cat /etc/systemd/system/trading-app.service
[Unit]
Description=Trading Application

[Service]
Type=simple
User=trader
ExecStart=/opt/trading/app
Restart=no  <-- No automatic restart!

[Install]
WantedBy=multi-user.target

The service has Restart=no - no automatic restart configured.

Solution

Basic Auto-Restart Configuration

# /etc/systemd/system/trading-app.service
[Unit]
Description=Trading Application
After=network.target

[Service]
Type=simple
User=trader
WorkingDirectory=/opt/trading
ExecStart=/opt/trading/app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Restart Policy Options

Advanced Auto-Restart with Limits

# /etc/systemd/system/trading-app.service
[Unit]
Description=Trading Application
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=trader
Group=trader
WorkingDirectory=/opt/trading
ExecStart=/opt/trading/app
ExecReload=/bin/kill -HUP $MAINPID

# Restart policy
Restart=on-failure
RestartSec=10

# Restart limits (prevents infinite restart loops)
StartLimitIntervalSec=300
StartLimitBurst=5

# Timeout configuration
TimeoutStartSec=30
TimeoutStopSec=30
TimeoutAbortSec=10

# Watchdog (restart if not responding)
WatchdogSec=60

# Environment
Environment="APP_ENV=production"
EnvironmentFile=/opt/trading/.env

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=trading-app

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/trading/data /var/log/trading

[Install]
WantedBy=multi-user.target

Restart with Backoff

# Exponential backoff for restarts
[Service]
Restart=on-failure
RestartSec=5
RestartMaxDelaySec=300
RestartSteps=5

Restart on Specific Exit Codes

# Custom restart policy
[Service]
Restart=on-failure
RestartForceExitStatus=SIGTERM SIGKILL 1 2 3
RestartPreventExitStatus=0 1 2

Service with Dependencies

# /etc/systemd/system/trading-app.service
[Unit]
Description=Trading Application
After=network.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service
BindsTo=postgresql.service

[Service]
Type=simple
User=trader
ExecStart=/opt/trading/app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Apply Changes

# Reload systemd
$ sudo systemctl daemon-reload

# Restart the service
$ sudo systemctl restart trading-app

# Verify restart policy
$ sudo systemctl show trading-app | grep -E "^(Restart|RestartSec|StartLimit)"
Restart=on-failure
RestartSec=5s
StartLimitBurst=5
StartLimitIntervalUSec=5min

# Check status
$ sudo systemctl status trading-app
● trading-app.service - Trading Application
     Loaded: loaded (/etc/systemd/system/trading-app.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2025-05-15 15:00:00 WET; 5s ago
   Main PID: 12345 (app)

Monitor Restarts

# View restart history
$ journalctl -u trading-app | grep -E "(Started|Stopped|Failed)"
May 15 14:30:00 server systemd[1]: Stopped Trading Application.
May 15 14:30:05 server systemd[1]: Started Trading Application.
May 15 15:00:00 server systemd[1]: Started Trading Application.

# Count restarts today
$ journalctl -u trading-app --since today | grep "Started" | wc -l
3

Reset Failed State

# Reset failed state to allow restart
$ sudo systemctl reset-failed trading-app

# Restart after failure
$ sudo systemctl start trading-app

Lessons Learned

1. Use Restart=on-failure for most services - it restarts on errors but not on clean shutdown.
2. Set RestartSec to prevent rapid restart loops.
3. Configure StartLimitBurst to prevent infinite restart attempts.
4. Use WatchdogSec to detect hung processes that are not responding.
5. Monitor restart frequency - frequent restarts indicate an underlying issue.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.