README_DISK_SPACE.md

Disk Space Management

Overview

The forge-server can accumulate cache files over time, leading to disk space issues. This document describes the disk space monitoring and cleanup mechanisms.

Current Disk Usage

Critical Threshold: 90%
Warning Threshold: 80%

Cleanup Mechanisms

1. Automatic Cache Cleanup (deleter.js)

• Frequency: Every 5 minutes
• Action: Removes cache files older than 7 days
• Location: /tmp/cache/ and /tmp/forgerc_sites/

2. Disk Space Cleanup Script

Location: scripts/disk-space-cleanup.sh

Features:

• Monitors disk usage on root filesystem (/)
• Performs routine cleanup of cache files older than 7 days
• Aggressive cleanup when disk usage exceeds 90%:
  • Removes cache files older than 1 day
  • Cleans old log files (30+ days)
  • Truncates large Docker log files
• Sends Slack notifications for warnings and critical alerts

Manual Execution:

sudo /app/scripts/disk-space-cleanup.sh

3. Systemd Timer (Recommended)

To run the cleanup script automatically every hour:

# Copy service and timer files
sudo cp systemd/disk-space-cleanup.service /etc/systemd/system/
sudo cp systemd/disk-space-cleanup.timer /etc/systemd/system/

# Enable and start the timer
sudo systemctl daemon-reload
sudo systemctl enable disk-space-cleanup.timer
sudo systemctl start disk-space-cleanup.timer

# Check status
sudo systemctl status disk-space-cleanup.timer
sudo systemctl list-timers disk-space-cleanup.timer

Monitoring

Check Current Disk Usage

df -h /

Check Cache Directory Size

du -sh /tmp/cache/
du -sh /tmp/forgerc_sites/

View Cleanup Logs

tail -f /var/log/disk-space-cleanup.log

Check Systemd Timer Status

sudo systemctl status disk-space-cleanup.timer
sudo journalctl -u disk-space-cleanup.service -f

Emergency Cleanup

If disk space is critically low (90%+), you can manually trigger aggressive cleanup:

# Run the cleanup script
sudo /app/scripts/disk-space-cleanup.sh

# Or manually clean old cache files
find /tmp/cache -type f -mtime +1 -delete
find /tmp/forgerc_sites -type f -mtime +1 -delete

# Clean empty directories
find /tmp/cache -type d -empty -delete
find /tmp/forgerc_sites -type d -empty -delete

Configuration

Adjust Cleanup Frequency

Edit deleter.js:

const CLEAR_INTERVAL = 5000*60; // 5 minutes (in milliseconds)
const CACHE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1000; // 7 days

Adjust Script Thresholds

Edit scripts/disk-space-cleanup.sh:

DISK_USAGE_THRESHOLD=80  # Alert at 80%
CRITICAL_THRESHOLD=90     # Critical at 90%
CACHE_MAX_AGE_DAYS=7      # Remove cache files older than 7 days

Adjust Systemd Timer Frequency

Edit systemd/disk-space-cleanup.timer:

OnUnitActiveSec=1h  # Change to desired frequency (e.g., 30m, 2h)

Troubleshooting

Disk Still Full After Cleanup

1. Check for large log files:

   find /var/log -type f -size +100M -ls

2. Check Docker container logs:

   docker ps
   docker logs <container-id> --tail 100

3. Check for large files in /tmp:

   find /tmp -type f -size +100M -ls

4. Check system logs:

   journalctl --disk-usage

Cleanup Script Not Running

1. Check script permissions:

   ls -l /app/scripts/disk-space-cleanup.sh
   # Should be executable: -rwxr-xr-x

2. Check systemd timer:

   sudo systemctl status disk-space-cleanup.timer

3. Check for errors:

   sudo journalctl -u disk-space-cleanup.service -n 50

Best Practices

1. Monitor Regularly: Set up alerts for disk usage above 80%
2. Regular Cleanups: Run cleanup script at least once per hour
3. Log Rotation: Ensure log rotation is configured for application logs
4. Docker Logs: Configure Docker log rotation to prevent log files from growing too large
5. Cache Management: Monitor cache directory size and adjust cleanup frequency if needed

Related Files

• deleter.js - Automatic cache cleanup
• scripts/disk-space-cleanup.sh - Manual cleanup script
• systemd/disk-space-cleanup.service - Systemd service file
• systemd/disk-space-cleanup.timer - Systemd timer file