tooolbox/blog/Linux Network Troubleshooting: ping, traceroute, and netstat
troubleshooting2024-02-15·11 min·323/348

Linux Network Troubleshooting: ping, traceroute, and netstat

A systematic approach to troubleshooting network issues on Linux using ping, traceroute, netstat, ss, and other diagnostic tools.

Introduction

Network issues are among the most common problems on Linux servers. Applications fail to connect, services become unreachable, and DNS stops resolving. A systematic approach to network troubleshooting saves time and prevents misdiagnosis. This post covers the essential network diagnostic tools and a methodology for isolating problems.

Environment

The examples use Ubuntu 22.04 LTS with a web server that is experiencing intermittent connectivity issues to external APIs.

ip addr show
# 1: lo: 
#     inet 127.0.0.1/8 scope host lo
# 2: eth0: 
#     inet 192.168.1.50/24 brd 192.168.1.255 scope global eth0

Problem

The web application intermittently fails to connect to an external API:

curl -v https://api.example.com/health
# * Trying 104.21.32.45:443...
# * connect to 104.21.32.45 port 443 failed: Connection timed out
# * Failed to connect to api.example.com port 443: Connection timed out

Analysis

Network troubleshooting follows the OSI model from bottom to top:

  1. Physical/Data Link: Is the interface up?
  2. Network: Can we reach the destination?
  3. Transport: Is the port accessible?
  4. Application: Is the service responding?

Solution

1. Check interface status

ip link show
# 2: eth0:  mtu 1500
#     link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff

# Check IP addresses
ip addr show eth0
# inet 192.168.1.50/24 brd 192.168.1.255 scope global eth0

# Check for errors
ip -s link show eth0

2. Test basic connectivity

# Ping gateway
ping -c 4 192.168.1.1
# PING 192.168.1.1: 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.5ms

# Ping external host
ping -c 4 8.8.8.8
# PING 8.8.8.8: 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=12.3ms

3. Trace route to identify breaks

traceroute api.example.com
#  1  gateway (192.168.1.1)  0.5ms
#  2  isp-router (10.0.0.1)  5.2ms
#  3  * * *
#  4  * * *
#  5  api.example.com (104.21.32.45)  15.6ms

Packet loss at hop 3-4 suggests an issue between the ISP and the destination.

4. Check DNS resolution

# Basic DNS lookup
nslookup api.example.com
# Server:    8.8.8.8
# Address:   8.8.8.8#53
# Non-authoritative answer:
# Name:    api.example.com
# Address: 104.21.32.45

# Detailed DNS lookup
dig api.example.com
# ;; ANSWER SECTION:
# api.example.com.    300    IN    A    104.21.32.45

# Check DNS servers
cat /etc/resolv.conf
# nameserver 8.8.8.8
# nameserver 8.8.4.4

5. Check port connectivity

# Test specific port
nc -zv api.example.com 443
# Connection to api.example.com 443 port [tcp/https] succeeded!

# Test multiple ports
nc -zv api.example.com {80,443,8080}
# Connection to api.example.com 80 port [tcp/http] succeeded!
# Connection to api.example.com 443 port [tcp/https] succeeded!
# nc: connect to api.example.com port 8080 (tcp) failed: Connection refused

6. Check listening services

# Using ss (modern replacement for netstat)
ss -tlnp
# State    Recv-Q Send-Q  Local Address:Port  Peer Address:Port
# LISTEN   0      128     0.0.0.0:80          0.0.0.0:*
# LISTEN   0      128     0.0.0.0:443         0.0.0.0:*
# LISTEN   0      128     [::]:80             [::]:*

# Using netstat (legacy)
netstat -tlnp

7. Check established connections

# Active connections
ss -tnp state established
# ESTAB  0  0  192.168.1.50:43210  104.21.32.45:443

# Count connections per state
ss -s
# Total: 1234
# TCP:   1200 (estab 800, closed 50, orphaned 200, timewait 300)

8. Check firewall rules

# Check iptables
sudo iptables -L -n -v

# Check UFW status
sudo ufw status verbose

# Check firewalld
sudo firewall-cmd --list-all

9. Check routing table

ip route show
# default via 192.168.1.1 dev eth0 proto dhcp metric 100
# 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.50

# Check for specific destination
ip route get 104.21.32.45
# 104.21.32.45 via 192.168.1.1 dev eth0 src 192.168.1.50

10. Monitor network traffic

# Capture packets
sudo tcpdump -i eth0 host api.example.com
# 14:30:01 IP 192.168.1.50.43210 > 104.21.32.45.443: Flags [S], seq 12345
# 14:30:01 IP 104.21.32.45.443 > 192.168.1.50.43210: Flags [S.], seq 67890

# Filter by port
sudo tcpdump -i eth0 port 443 -n

11. Check bandwidth

# Install iftop
sudo apt-get install iftop

# Monitor bandwidth
sudo iftop -i eth0

12. Test with curl verbose

curl -v --connect-timeout 10 https://api.example.com/health
# * Trying 104.21.32.45:443...
# * Connected to api.example.com (104.21.32.45) port 443
# * SSL connection using TLSv1.3
# > GET /health HTTP/2
# < HTTP/2 200

Lessons Learned

Network troubleshooting is methodical: start from the physical layer and work up. Use ping for basic connectivity, traceroute for path analysis, and ss/netstat for local state. Always check DNS, firewall rules, and routing before assuming a network issue. And capture packets with tcpdump for deep analysis when other tools are insufficient.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.