No Space Left on Device on Linux: How to Fix It

Quick answer: Run df -h to identify which partition is at 100%. Use du -h --max-depth=1 /path | sort -rh | head -20 to drill into it. The three fastest wins are: vacuum journald logs with journalctl --vacuum-size=500M, clear the package cache with apt clean, and restart whichever process is holding deleted files open — find it with lsof +L1.

Start with df -h — no guessing

The error No space left on device tells you nothing about where the problem is. First, figure out which filesystem is actually full:

df -h

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G   39G     0  100% /
/dev/sdb1       500G   80G  420G   16% /data
tmpfs           7.8G  1.1G  6.7G   14% /dev/shm

Look at the Use% column. Find the row at 100% and note its mount point — that's what you're dealing with. Don't skip this step. I've watched people spend half an hour deleting things on /data when /var was full the whole time.

One thing df -h won't tell you: if the numbers seem wrong — say, df shows the disk full but du totals don't add up — you likely have deleted files still held open by a process. Jump ahead to the ghost-space section if that's the situation.

How to find large files and directories with df and du

Once you know which filesystem is full, drill into it with du. Start one level deep from the mount point:

sudo du -h --max-depth=1 /var 2>/dev/null | sort -rh | head -20

2>/dev/null silences permission errors. sort -rh sorts by human-readable size, largest first. Identify the biggest directory in the output, then run the same command inside it. Keep drilling until you've found the actual source. On most Linux servers the culprits are /var/log, /var/cache, or /var/lib/docker.

If you want to hunt individual large files rather than directory totals:

sudo find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null

The -xdev flag is important — it stops find from crossing into other mounted filesystems, which would give you a misleading picture on a multi-partition setup.

For interactive navigation, ncdu is worth the install. It builds a sortable directory tree you can walk with arrow keys:

# Debian/Ubuntu
sudo apt install ncdu

# RHEL/CentOS/Fedora
sudo dnf install ncdu

ncdu /

On a server with a large filesystem, ncdu will feel faster than re-running du repeatedly.

How to clear journald logs and reclaim gigabytes fast

On any systemd server that's been running for a few months, journald is usually the first thing I check. It can silently grow to several gigabytes. Check current usage:

journalctl --disk-usage

You might see something like: Archived and active journals take up 4.2G in the file system. If it's well over a few hundred megabytes and you're not actively debugging a problem, trim it:

# Keep only the most recent 500 MB of logs
sudo journalctl --vacuum-size=500M

# Or: remove everything older than 14 days
sudo journalctl --vacuum-time=14d

Both flags only remove archived journal files — the active journal is untouched. To flush the active journal too, rotate it first, then vacuum:

sudo journalctl --rotate
sudo journalctl --vacuum-size=500M

Doing this once and forgetting about it means you're back here in six months. Add a permanent size cap by creating /etc/systemd/journald.conf.d/size.conf:

[Journal]
SystemMaxUse=500M

Then reload the daemon:

sudo systemctl restart systemd-journald

From that point journald enforces the limit itself. On a small VM with under 20 GB of root disk, set it to 200M. On a production server where you actually read logs, 1G is more practical. Official config reference: journald.conf man page.

Clear the package cache

On Debian and Ubuntu, apt stores every downloaded .deb in /var/cache/apt/archives/. On a server that gets updated regularly, this accumulates silently.

# Removes all cached .deb files — safe, doesn't touch installed packages
sudo apt clean

# Removes dependency packages no longer needed by anything installed
sudo apt autoremove

apt clean is always safe on a production system. If you reinstall something after running it, apt re-downloads the package. That's the only cost. Run it freely.

On RHEL, CentOS 8+, or Fedora:

sudo dnf clean all

On older CentOS 7:

sudo yum clean all

While you're here, check for accumulated old kernels:

# List installed kernel packages
dpkg --list | grep linux-image

# Show currently running kernel — don't remove this one
uname -r

On Ubuntu, apt autoremove usually handles old kernels automatically. If it doesn't, remove specific old versions manually with apt purge linux-image-X.X.X-XX-generic, but always verify you're not removing the running kernel first.

The ghost-space problem: deleted files still held open

This one catches everyone at least once. You delete a 15 GB log file, run df -h, and the number barely moves. The reason: the process that opened the file — nginx, a JVM, a database — still holds an open file descriptor pointing to it. Linux won't release the blocks until that handle is closed. The file is gone from the directory listing but still consuming space on disk.

Find those files with:

sudo lsof +L1

+L1 lists files with a link count below 1 — meaning no directory entry remains, but the descriptor is still live. Output looks like:

COMMAND    PID  USER  FD   TYPE DEVICE         SIZE NLINK  NODE NAME
java     49097   app  77w   REG  253,6  33955068440     0  1283397 /opt/app/logs/server.log (deleted)

That (deleted) marker next to a 33 GB file tells you exactly what happened. The fix is to restart the offending process:

sudo systemctl restart your-service-name

Space comes back immediately. If restarting is off the table — say, a production database at 2am — you can truncate the file in-place through /proc without killing the process:

sudo truncate -s 0 /proc/49097/fd/77

Replace 49097 with the PID and 77 with the file descriptor number from the lsof output. Use this carefully. Some applications — especially databases writing to WAL or transaction logs — can end up in a corrupted state if an open file is truncated under them. Prefer a controlled restart when you can schedule one.

Inode full: when df -h shows free space but writes still fail

An inode is the metadata record for a file — permissions, timestamps, pointer to data blocks. Ext4 allocates a fixed number of inodes when the filesystem is formatted, and you can exhaust them long before you fill the data blocks. The result is the exact same error message, with no obvious cause.

Check inode usage:

df -i

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      2621440 2621440       0  100% /

If IUse% hits 100% while df -h still shows available data blocks, you've confirmed inode exhaustion. The usual culprits: PHP session files, mail queue entries, temp files from a misbehaving application, or any process that creates enormous numbers of tiny files.

Find the directory hoarding them:

sudo find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -rn | head -20

This groups file counts by third-level directory path. A mail queue or session directory with hundreds of thousands of entries will stand out immediately. Delete the unnecessary files and inodes free up. You can't add more inodes to an existing ext4 filesystem without reformatting — so fix whatever is generating the excess files once you've cleared the backlog.

The ext4 reserved-blocks trick for data partitions

By default, ext4 reserves 5% of a filesystem's blocks for root-owned processes. The rationale is that system daemons like syslogd can keep writing even when a regular user fills the disk. On your root (/) filesystem, leave this alone — it's the safety net that lets you log in as root and fix things. On a dedicated data partition, it's just wasted space: 5% of a 2 TB drive is 100 GB.

Check what's currently reserved:

sudo tune2fs -l /dev/sdb1 | grep 'Reserved block'

Lower it to 1% on a data-only partition:

sudo tune2fs -m 1 /dev/sdb1

Or zero it out entirely if no system processes write to it:

sudo tune2fs -m 0 /dev/sdb1

This works on a live, mounted filesystem — no unmounting required. It applies to ext3 and ext4 only. XFS and btrfs don't have an equivalent reserved-blocks mechanism. Full flag reference: tune2fs man page.

Linux disk full fix: quick-reference table

Problem	Command
Which partition is full?	df -h
Biggest directories in a path	sudo du -h --max-depth=1 /path | sort -rh | head -20
Find individual files over 100 MB	sudo find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null
Interactive disk browser	ncdu /
How much space journald is using	journalctl --disk-usage
Vacuum journald logs	sudo journalctl --vacuum-size=500M
Clear apt package cache (Debian/Ubuntu)	sudo apt clean && sudo apt autoremove
Clear DNF cache (RHEL/CentOS 8+/Fedora)	sudo dnf clean all
Find deleted files still held open	sudo lsof +L1
Check inode usage	df -i
Check ext4 reserved blocks	sudo tune2fs -l /dev/sdX | grep Reserved
Reduce ext4 reserved to 1% (data partitions only)	sudo tune2fs -m 1 /dev/sdX

FAQ: no space left on device

Why does df show the disk is full but du doesn't account for all the space?

Almost always: deleted files still held open by a running process. du traverses the directory tree and only counts files that have directory entries. df reports actual filesystem block usage, which includes blocks claimed by open-but-unlinked files. Run sudo lsof +L1 to identify them, then restart the holding process.

Can I get "no space left on device" even when df -h shows free space?

Yes — two common causes. First, inode exhaustion: run df -i and check whether IUse% is at 100%. Second, disk quotas: run quota -v or check your /etc/fstab for quota mount options. Both produce the exact same error message even when data blocks are available.

How do I fix "no space left on device" on a Docker host?

Docker accumulates dangling images, stopped containers, unused volumes, and build cache — often tens of gigabytes on a CI host. Check usage first: docker system df. Then prune: docker system prune -a removes all unused images and stopped containers. If you want to keep named volumes intact, don't add the --volumes flag. Run this regularly on any host running automated builds.

Is it safe to run apt clean on a production server?

Yes. It only removes cached .deb installer files from /var/cache/apt/archives/. Installed packages are completely unaffected. The only consequence is that if you need to reinstall a package later, apt downloads it again from the repository. That's a non-issue on a server with internet access.

How do I stop this from happening again?

Three things make the biggest difference: set SystemMaxUse in /etc/systemd/journald.conf.d/size.conf to cap journal growth; make sure logrotate is configured and actually running for your application logs (/etc/logrotate.d/); and add a disk-space alert. A cron job that runs df -h | awk '$5 > "85%"' and mails you is enough to stop getting surprised. For production systems, a Prometheus alert on node_filesystem_free_bytes is a better long-term answer.