Two of my pylons apps had session files that slowly ran away on me. The session files don’t consume much capacity however the shear quantity of them caused my inode usage to hit 100%.
Had I properly configured Munin’s email alerting this issue would have been identified well before it was a problem.
Want to know what alerted me to the problem? G Webmaster’s tools claimed it could not read my robots.txt on a few of my sites… After investigating I learned the site was down. Checking the Apache error logs pointed me to disk space issues.
df -ha reported everything was fine, however
df -hi reported 100% inode usage! At this point I started looking to cache and log locations to find lots of files, which lead me to my pylons web applications data/sessions directories.
Delete the session cache tree directories and allow the applications to rebuild them.
Todo: move /www off the root disk partition. This issue could have been much worse if I was unable to boot or login to remedy. Moving /www off root should prevent the web server from effecting the systems ability to boot.