Linux Inotify: File System Monitoring for Automation

Introduction

Monitoring file system changes in real time is essential for automation. Whether you need to trigger deployments when code changes, process uploaded files, or watch log directories, inotify provides an efficient kernel-level mechanism. This post covers using inotify-tools on Linux for file system monitoring and automation.

Environment

The examples use inotify-tools 3.22 on Ubuntu 22.04 LTS, monitoring directories for file creation, modification, and deletion.

dpkg -l | grep inotify
# ii  inotify-tools    3.22-1build1    amd64    inotify-wait/notify tools

# Check inotify support
cat /proc/sys/fs/inotify/max_user_watches
# 524288

Problem

You need to automate tasks based on file system events:

• Process files when they appear in a directory
• Trigger deployments when code changes
• Monitor log files for specific patterns
• Sync files between directories in real time

Analysis

inotify monitors file system events at the kernel level:

The inotifywait command from inotify-tools provides a simple interface for monitoring:

inotifywait --help
# inotifywait [OPTION...]  [...]
# -m, --monitor    Listen for events continuously
# -r, --recursive  Watch directories recursively
# -e, --event LIST  Events to watch

Solution

1. Basic directory monitoring

# Wait for any event in a directory
inotifywait /data/uploads/
# Setting up watches.
# Watches established.
# /data/uploads/ CREATE file.txt

2. Monitor specific events

# Monitor only file creation
inotifywait -e create /data/uploads/

# Monitor creation and modification
inotifywait -e create -e modify /data/uploads/

# Monitor all events
inotifywait -m -r -e create,modify,delete,move /data/uploads/

3. Real-time monitoring with continuous output

# Monitor continuously with formatted output
inotifywait -m -r --format '%w%f %e %T' --timefmt '%Y-%m-%d %H:%M:%S' /data/uploads/
# /data/uploads/file1.txt CREATE 2024-04-15 10:00:00
# /data/uploads/file1.txt CLOSE_WRITE 2024-04-15 10:00:01

4. Trigger actions on events

#!/bin/bash
# watch-and-process.sh

inotifywait -m -r -e create --format '%w%f' /data/uploads/ | while read file; do
  echo "Processing: $file"
  python3 /scripts/process_file.py "$file"
  mv "$file" /data/processed/
done

5. Monitor log files for patterns

# Watch for new log entries
inotifywait -m -e modify /var/log/nginx/access.log | while read file event; do
  tail -1 /var/log/nginx/access.log | grep "500" && \
    echo "ERROR: 500 response detected" | mail -s "Alert" admin@example.com
done

6. Sync directories in real time

#!/bin/bash
# sync-directories.sh

inotifywait -m -r -e create,modify,delete /source/ | while read dir event file; do
  case $event in
    CREATE|MODIFY)
      rsync -av --delete /source/ /dest/
      ;;
    DELETE)
      rm -f "/dest/$file"
      ;;
  esac
done

7. Monitor with exclude patterns

# Ignore temporary files
inotifywait -m -r \
  --exclude '\.(swp|tmp|bak)$' \
  /data/uploads/

8. Use inotify with systemd

# /etc/systemd/system/file-watcher.service
[Unit]
Description=File System Watcher
After=network.target

[Service]
Type=simple
ExecStart=/opt/scripts/watch-and-process.sh
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

9. Adjust inotify limits

# Check current limits
cat /proc/sys/fs/inotify/max_user_watches
# 524288

# Increase if needed
echo 1048576 | sudo tee /proc/sys/fs/inotify/max_user_watches

# Persist
echo "fs.inotify.max_user_watches=1048576" | sudo tee -a /etc/sysctl.d/99-inotify.conf

10. Monitor multiple directories

inotifywait -m -r \
  /data/uploads/ \
  /data/processed/ \
  /var/log/nginx/ \
  --format '%w %e %f'

11. Use inotify for deployment automation

#!/bin/bash
# auto-deploy.sh

WATCH_DIR="/opt/myapp/src"
DEPLOY_CMD="cd /opt/myapp && git pull && npm install && pm2 restart all"

inotifywait -m -r -e modify "$WATCH_DIR" | while read dir event file; do
  echo "Change detected in $file, deploying..."
  eval "$DEPLOY_CMD"
  sleep 5  # Debounce rapid changes
done

12. Process files as they arrive

#!/bin/bash
# process-uploads.sh

UPLOAD_DIR="/data/uploads"
PROCESSED_DIR="/data/processed"
LOG_FILE="/var/log/upload-processor.log"

inotifywait -m -e close_write "$UPLOAD_DIR" | while read dir event file; do
  echo "$(date): Processing $file" >> "$LOG_FILE"
  
  # Validate file
  if file "$UPLOAD_DIR/$file" | grep -q "PDF"; then
    # Process PDF
    python3 /scripts/extract_pdf.py "$UPLOAD_DIR/$file"
  fi
  
  # Move to processed
  mv "$UPLOAD_DIR/$file" "$PROCESSED_DIR/"
  echo "$(date): Completed $file" >> "$LOG_FILE"
done

Lessons Learned

inotify is the most efficient way to monitor file system changes on Linux. Use -m for continuous monitoring and --format for structured output. Always adjust max_user_watches if monitoring many directories. And debounce rapid file changes with a short sleep to avoid processing partial writes.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.