I got surprised that some of my containers stopped and then cannot be started. I got a strange full size errors.
This was shocking because I did remember my SDD usage was about ~30% of the size.
So I decided to see what is going on. First I wanted to see if it is really full or some docker bug.
df -hthat gives me the answer
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 31M 1.6G 2% /run
/dev/sda2 392G 392G 0 100% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 922M 221M 638M 26% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/1000Now it is 100% occupied. But what is really taking this space. I suspected something related to some log files going crazy.
But because I relay on docker much on this server I thought maybe some volume there went crazy so I checked
docker system df -vreturned
Images space usage:
REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS
ghost alpine 6b64a14003a3 4 days ago 390.8MB 5.575MB 385.2MB 1
linuxserver/calibre-web latest 7a30b9d2cdef 2 weeks ago 619MB 0B 619MB 1
linuxserver/thelounge latest aad7b72045e6 2 weeks ago 251.1MB 24.85MB 226.3MB 1
mongo 4.0 362f994056cd 2 weeks ago 430.3MB 0B 430.3MB 1
vaultwarden/server alpine 6e63163bb391 2 weeks ago 90.12MB 5.575MB 84.54MB 1
pihole/pihole latest 52916ce4626f 2 weeks ago 303.8MB 80.41MB 223.3MB 1
ghcr.io/linuxserver/endlessh latest a05ef23bdc45 2 weeks ago 25.14MB 24.85MB 288.2kB 1
linuxserver/nextcloud latest 33d87a7432d3 2 weeks ago 424MB 0B 424MB 1
ttionya/vaultwarden-backup latest 90f66fc648b1 3 weeks ago 61.73MB 5.575MB 56.15MB 1
postgres 12-alpine b76727572d69 3 weeks ago 208.9MB 5.575MB 203.3MB 1
amir20/dozzle latest c3e822600f0c 3 weeks ago 10.11MB 0B 10.11MB 1
ghcr.io/linuxserver/heimdall latest 621c19471beb 3 weeks ago 94.47MB 0B 94.47MB 1
authelia/authelia latest 0d23447e4684 3 weeks ago 48.91MB 5.575MB 43.34MB 1
redis 5.0 972767e7d959 4 weeks ago 109.7MB 80.41MB 29.24MB 1
qmcgaw/gluetun latest dc68aaf68f41 5 weeks ago 29.89MB 5.575MB 24.31MB 1
linkace/linkace simple fd21d5672147 5 weeks ago 562.6MB 5.586MB 557MB 1
melashri/keeweb latest 50e24f83e4d5 5 weeks ago 33.21MB 5.575MB 27.64MB 1
melashri/snibox latest ecae3f3ae857 5 weeks ago 1.159GB 5.605MB 1.153GB 1
searx/searx latest e38001222d91 8 weeks ago 186.3MB 5.575MB 180.8MB 1
melashri/overleaf latest 3d058273521c 2 months ago 7.66GB 0B 7.66GB 1
jc21/nginx-proxy-manager latest 7c775dbb91f2 2 months ago 921.1MB 0B 921.1MB 1
erikvl87/languagetool latest 54ca3f4092d5 2 months ago 550.2MB 0B 550.2MB 1
klutchell/unbound latest 82cb65cda6f2 2 months ago 13.94MB 0B 13.94MB 1
sissbruecker/linkding latest 0315dfc0e06f 2 months ago 202.3MB 0B 202.3MB 1
xavierh/goaccess-for-nginxproxymanager develop efd4755d581e 3 months ago 98.44MB 5.586MB 92.85MB 1
linuxserver/librespeed latest b6d632ca3b2a 3 months ago 84.76MB 0B 84.76MB 1
ekzhang/rustpad latest 3f043f0d795a 3 months ago 14.64MB 0B 14.64MB 1
freshrss/freshrss latest 7d426d2491cd 4 months ago 215.6MB 0B 215.6MB 1
containrrr/watchtower latest 333de6ea525a 4 months ago 16.92MB 0B 16.92MB 1
weejewel/wg-easy latest 76142e771591 4 months ago 141.2MB 0B 141.2MB 1
melashri/cocalc latest 80f058aa8494 5 months ago 28.18GB 0B 28.18GB 1
pawelmalak/flame latest 079642c721f9 6 months ago 182.8MB 5.605MB 177.2MB 1
portainer/portainer-ee 2.10.0 12043031b306 7 months ago 261MB 0B 261MB 1
keyoxide/keyoxide stable 4d4931dfb98c 7 months ago 232.7MB 118MB 114.6MB 1
keyoxide/doip-proxy stable d24880ab8402 7 months ago 294.1MB 118MB 176MB 1
lscr.io/linuxserver/calibre-web latest e5aa9acc9992 8 months ago 568.2MB 0B 568.2MB 0
ghcr.io/linuxserver/wireshark latest 031cfee14376 9 months ago 1.209GB 0B 1.209GB 1
ghcr.io/linuxserver/webtop latest 8d43f37852ca 10 months ago 1.127GB 0B 1.127GB 1
qmcgaw/2fauth latest 773f191b660b 10 months ago 195.2MB 69.26MB 126MB 1
hlsiira/atheos latest 93a6b433d679 10 months ago 419.7MB 0B 419.7MB 1
mltooling/ml-workspace 0.13.2 ddee8f9abca3 11 months ago 10.07GB 0B 10.07GB 1
afian/filerun latest 058b811e09cb 11 months ago 1.07GB 69.26MB 1GB 1
archivebox/archivebox master a8da1f0258b3 13 months ago 1.235GB 0B 1.235GB 2
mariadb 10.1 895244a22f37 13 months ago 351.9MB 0B 351.9MB 1
melashri/latex latest 15b1cf2ae7f5 14 months ago 1.851GB 0B 1.851GB 1
latex latest 1900b858ef95 14 months ago 887.4MB 0B 887.4MB 1
ghcr.io/linuxserver/cloud9 latest 57f0ad3c6a42 14 months ago 919.1MB 0B 919.1MB 1
lovasoa/wbo latest 52fc2d2b1aa0 14 months ago 391.7MB 0B 391.7MB 1
selfhostedpro/yacht latest a8145c3aefe5 15 months ago 406.3MB 0B 406.3MB 1
amruthpillai/reactive-resume v1 74f750151508 16 months ago 60.09MB 0B 60.09MB 1
screego/server 1.1.2 f5abd32dfba0 17 months ago 13.78MB 0B 13.78MB 1
mltooling/ml-workspace-r 0.12.1 c23fa1157f55 17 months ago 11.8GB 0B 11.8GB 1
metabase/metabase v0.37.0.2 868fb1266033 19 months ago 387.8MB 0B 387.8MB 1
dockupdater/dockupdater latest 859d6e8ec78b 19 months ago 77.99MB 0B 77.99MB 1
jc21/mariadb-aria 10.4 174fbfd81bac 2 years ago 216.3MB 0B 216.3MB 1
nurdism/neko firefox 03d214359368 2 years ago 541.2MB 0B 541.2MB 1
dclong/jupyterhub-beakerx latest 1cab2aaed76f 2 years ago 3.988GB 0B 3.988GB 1
barttc/dpaste latest 8bbaac192f0f 2 years ago 1.003GB 0B 1.003GB 1
dm3ll3n/shinystudio latest 0737251fe6ec 2 years ago 4.976GB 0B 4.976GB 1
This looks normal and although I have too many, still nothing out of ordinary.
Okay, now lets check the ordinary suspects /tmp and /var/log
I didn't bother with checking /tmp I went and deleted it right away
sudo rm -r /tmpbut again df -h gets me
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 112M 1.5G 7% /run
/dev/sda2 392G 392G 0 100% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 922M 221M 638M 26% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/1000I was harsh with /tmp sorry you. But I still remember that on fermilab shared machines pool
you caused me trouble before in front of ~300 people. So I still hate you.
Now I decided that I will not take a guess and lets go and see who is making this mess.
I went to / and ran
sudo du -hc --max-depth=1 /This checks out folders size. I got
2.0G /root
0 /dev
16K /lost+found
51M /yacht
0 /proc
2.0G /usr
4.0K /lib64
0 /sys
2.3G /volume1
5.1G /filerun
409G /var
12M /bin
2.0M /sharelatex_data
16K /redis_data
4.0K /syncserver
4.0K /media
88M /data
199M /portainer
4.0K /srv
4.0K /mnt
501M /mongo_data
184K /bw-data
254M /path
24K /tmp
4.0K /sharelatex_packages
172M /run
180M /opt
219M /boot
15M /sbin
8.0K /ngrams
7.8M /etc
4.9G /home
2.9M /volum1
1.4G /lib
12K /phome
430G /
430G totalNow as I have guessed first, this is 409G /var so probably it is /var/log but lets check this
Another way of doing that using find with more flexbility
find /var -maxdepth 1 -type d -exec du -smh '{}' \;409G /var
7.4M /var/cache
4.0K /var/local
4.0K /var/mail
28K /var/tmp
4.0K /var/opt
24K /var/spool
408G /var/lib
797M /var/log
988K /var/backupsWow, it seems that /var/log is innocent, my guess is wrong again. still I feel like it must be some log file problem.
But as we can see problem is withing /var/lib so lets go and see. Again lets cut to the chase and see what is taking space.
find /var/lib -maxdepth 1 -type d -exec du -smh '{}' \;who is the offender?
408G /var/lib
4.0K /var/lib/usb_modeswitch
4.0K /var/lib/ntpdate
235M /var/lib/mlocate
12K /var/lib/AccountsService
12K /var/lib/locales
2.1M /var/lib/containerd
8.0K /var/lib/ubuntu-release-upgrader
84K /var/lib/fail2ban
4.0K /var/lib/focalboard
4.0K /var/lib/python
56K /var/lib/libvirt
8.0K /var/lib/ureadahead
24K /var/lib/emacsen-common
4.0K /var/lib/unattended-upgrades
8.0K /var/lib/ispell
8.0K /var/lib/sudo
12K /var/lib/grub
28K /var/lib/pam
44K /var/lib/polkit-1
4.0K /var/lib/man-db
4.0K /var/lib/ntp
4.0K /var/lib/update-notifier
544K /var/lib/usbutils
4.0K /var/lib/os-prober
12K /var/lib/update-manager
4.0K /var/lib/udisks2
252M /var/lib/apt
52M /var/lib/dpkg
4.0K /var/lib/plymouth
4.0K /var/lib/cockpit
104K /var/lib/ucf
4.0K /var/lib/command-not-found
8.0K /var/lib/NetworkManager
16K /var/lib/ubuntu-advantage
407G /var/lib/docker
8.0K /var/lib/aspell
72M /var/lib/crowdsec
16K /var/lib/dictionaries-common
4.0K /var/lib/misc
408K /var/lib/systemd
60K /var/lib/PackageKit
8.0K /var/lib/dbus
12K /var/lib/private
4.0K /var/lib/dhcp
4.0K /var/lib/git
4.0K /var/lib/logrotate
8.0K /var/lib/vim
20K /var/lib/initramfs-toolsoh it is 407G /var/lib/docker so docker is that you after all? I was uh, okay maybe I wasn't wrong.
Okay lets see then what is the problem. is it a container there or image or a volume going crazy?
find /var/lib/docker -maxdepth 1 -type d -exec du -smh '{}' \;which thing is the problem?
16K /var/lib/docker/plugins
274G /var/lib/docker/containers
2.3G /var/lib/docker/volumes
4.0K /var/lib/docker/runtimes
251M /var/lib/docker/image
4.0K /var/lib/docker/swarm
4.0K /var/lib/docker/tmp
4.0K /var/lib/docker/trust
392K /var/lib/docker/network
132G /var/lib/docker/overlay2
92K /var/lib/docker/buildkitoh seems like containers and overlay are inflated but they are related. So which container is the problem. running
find /var/lib/docker/containers -maxdepth 1 -type d -exec du -smh '{}' \; gives me the problematic one. I use portainer so quickly search for it
and found it is Goaccess which is an open source real-time web log analyzer for ngnix reverse proxy. I remember I setup this but havn't used it in a long time.
It reads basically your ngnix log file which you mount the container volume to its location. It appears like there was a bug that created this mess. I removed the container and deleted the volume.
Problem solved and everything is fine.