When a server suddenly throws the dreaded “No space left on device” error, it’s not just an annoyance—it’s a critical operational failure. Services fail to write necessary logs, web applications crash because they can’t store session files, and databases refuse writes entirely. In the heat of a crisis, the immediate instinct is often panic, leading to rash commands that might break something else.
However, understanding why your disk filled up is more important than merely deleting random files. For most modern Linux deployments, space exhaustion rarely stems from the base operating system itself; rather, it’s due to the slow, relentless accumulation of variable data: logs that fail to rotate, temporary caches left behind by complex builds, or database binary logs that never expire.
This guide is designed for experienced systems administrators who need a structured, actionable workflow. We will move past generic advice and dive into specific mechanisms—from diagnosing inode exhaustion to safely vacuuming systemd journals and purging forgotten data blocks—ensuring you can not only solve the immediate crisis but also build guardrails against future outages.
Initial Diagnosis: Determining if You are Out of Bytes or Inodes
The single most common mistake when facing a full disk is assuming that “full” means that every byte has been written. This is often incorrect. Before running any cleanup commands, you must confirm the nature of the problem by checking two key metrics provided by the df command: bytes used and inodes used.
- Filesystem Usage (Bytes): Run
df -h. This tells you which specific mount point (e.g.,/,/var,/tmp) is near 95–100% capacity in terms of raw storage space. If this column shows high utilization, your disk has run out of bytes. - Inode Usage (File Count): Run
df -ih. Inodes are the structures that store information about files—metadata like owner, size, and location. Even if you have gigabytes of free space remaining, if a directory is filled with millions of tiny files (such as session keys, cache entries, or spam mail), your inodes can become exhausted. This scenario presents as “No space left on device” even when thedf -houtput shows available capacity.
If both metrics are high, you have a compound issue requiring dual-pronged cleanup. If only one is high, proceed with targeted cleanup for that specific resource type.
Taming the Log Beast: Controlling /var/log and Systemd Journals
Log files are perhaps the most frequent culprit in disk bloat. They are designed to record every event, error, and connection attempt—which is invaluable for debugging, but disastrous if left unbounded.
The Modern Journal (systemd-journald)
On modern distributions using systemd, logs are managed by the journal daemon (journalctl). Unlike traditional plain text files, these journals can grow to tens or even hundreds of GB over time if retention limits are not explicitly set. Do not simply delete the journal directory.
To safely manage space:
- Check Current Usage:
journalctl --disk-usage - Vacuum by Size (Recommended): To trim the logs while keeping a minimum amount, use
--vacuum-size=X. For example, to keep the size under 500MB:sudo journalctl --vacuum-size=500M - Vacuum by Time: To ensure you only retain logs from the last two days:
sudo journalctl --vacuum-time=2d
Traditional System Logs (/var/log)
For older or application-specific logs (like web server access logs, mail queues, etc.), the danger lies in failed log rotation or runaway applications.
- Truncating vs. Deleting: If a service is actively writing to a large file and you need immediate space, do not use
rm. The process will continue to write to the inode, and the operating system will report that the space hasn’t been reclaimed. Instead, usesudo truncate -s 0 /path/to/logfile. This empties the file contents without changing its file descriptor or size on disk. - Prevention: Always verify that your
logrotateconfiguration files are active and set appropriate rotation limits (e.g., keeping only 7 daily rotations, limiting maximum size).
Clearing Transient Waste: /tmp, Package Caches, and System Artifacts
Temporary directories (/tmp, /var/tmp) and package caches are designed to hold short-term data—data that should be ephemeral. When they accumulate junk or large artifacts from failed builds, they quickly become space hogs.
Package Repository Cleanup
When running system upgrades or installing new packages using a distribution’s package manager, the downloaded .deb or .rpm files accumulate in cache directories. These are safe to clear because they are merely local copies of already installed software.
- For Debian/Ubuntu:
sudo apt clean(removes archived packages) andsudo apt autoremove --purge(removes dependencies no longer needed). - For RedHat/Fedora/AlmaLinux: Use the equivalent commands for your package manager to clear downloaded metadata and obsolete packages.
Temporary Filesystems
While /tmp often defaults to a RAM-backed filesystem (tmpfs) which clears upon reboot, the persistent sibling, /var/tmp, can accumulate large files from long-running processes or complex builds (like Docker container build layers). Regularly inspecting this directory for stale data is critical.
Identifying Orphaned Data
If you suspect general system bloat but cannot pinpoint a location, use the du -sh /path command to audit usage across major subdirectories within /var. The -x flag is crucial here; it ensures that the disk usage calculation does not cross into other mounted filesystems, keeping your investigation focused on the intended partition.
Database Bloat Management: Purging Forgotten Binary Logs and Sessions
Databases are specialized consumers of disk space. While they handle data gracefully, certain operational features—which are excellent for recovery or replication—can become self-perpetuating sources of bloat if not managed with strict expiration policies.
Replication and Transactional Logs
Many robust database systems (like MySQL) enable binary logging (binlog) to track every write operation. This is essential for point-in-time recovery, but the logs are append-only and will grow indefinitely unless they are explicitly purged.
The fix requires administrative intervention within the database itself:
- Check Current Logs: Identify the location of the binary log directory (often under
/var/lib/mysql). - Purge Old Data: Use the native purge commands provided by your specific database engine, setting a reasonable time limit. For example, instructing the system to only retain logs for the last 7 days is a standard best practice.
Session and Cache Data
Database engines also create session files and cache indexes that contribute to bloat. Proper configuration of maximum connection limits, memory usage thresholds, and scheduled cleanup jobs are necessary preventative measures. Furthermore, applications that rely on external key-value stores (like Redis) must have their own eviction policies configured to prevent indefinite growth due to unused keys or expired data.
Advanced Triage: File Descriptors and Inode Exhaustion
If the previous steps haven’t yielded enough space, you are likely dealing with a more advanced issue involving resource management or inode exhaustion.
Deleted-but-Open Files
This is one of the most insidious problems. A process may delete a file (e.g., an old log file) but still hold an open file descriptor to it. From the OS’s perspective, the space occupied by that data block has not been released until the holding process exits or restarts.
To diagnose this, use lsof +L1. This command lists files opened by processes and specifically looks for “deleted” status indicators (+L1), allowing you to pinpoint which running services are responsible for consuming disk blocks that appear “gone.” The solution is often to gracefully restart the specific service holding the descriptor.
Inode Remediation
If df -ih shows 100% utilization, your issue is file count, not byte size. This usually points to poorly configured cache systems or spam/mail queues creating millions of tiny files. Since the problem is sheer volume, targeted cleanup (e.g., cleaning mail queues, clearing application-specific session directories) must be performed manually and carefully, as deleting large batches of small files can sometimes indicate a deeper systemic issue with file creation.
Conclusion
Running out of disk space is fundamentally a failure of maintenance policy, not typically a failure of the underlying hardware or operating system. The solution requires adopting a systematic approach: first, confirming whether the problem is byte-based or inode-based; second, implementing strict, automated retention policies for high-volume data sources like logs and database binary archives; and third, routinely clearing ephemeral caches and package artifacts. By proactively setting size limits on systemd journals, managing log rotation with established tools, and regularly auditing the growth patterns of your databases, you can transform disk space management from a reactive firefighting exercise into a stable, predictable element of system maintenance.
Frequently Asked Questions (FAQ)
Q: What is the critical difference between running out of bytes and inodes?
A: This confusion is common, but vital to diagnose correctly. Low disk space might mean you have run out of bytes (raw storage capacity), which is checked using df -h. Conversely, if a directory holds millions of tiny files, your system can exhaust its inodes, the metadata structures for those files—even if gigabytes of raw space remain. Always check both metrics to determine the true bottleneck.
Q: How should I safely manage massive logs created by systemd-journald?
A: Never simply delete the journal directory, as this can cause service issues. Instead, use specific commands to prune the data based on your needs. To limit space usage, run sudo journalctl --vacuum-size=X, or to ensure you only keep recent activity, use sudo journalctl --vacuum-time=Y.
Q: If a large file is actively being written to by an application and I need immediate space, what should I do?
A: Never use the standard rm command on a running log file, as the process will simply continue writing over the reclaimed space. The correct method for immediately freeing the inode while keeping the file structure intact is using the truncate command (e.g., sudo truncate -s 0 /path/to/file).
Q: What does “No space left on device” mean if my df -h output shows available capacity?
A: This error strongly suggests an inode exhaustion problem, meaning the system cannot create new file metadata structures for incoming files. It indicates that while you might have plenty of empty gigabytes, your filesystem has run out of pointers to track those files. In this scenario, cleanup must focus on deleting large numbers of small files in a specific directory.
