Isilon /var full usage at 100% but few files

I had this problem last week. The /var filesystem was full, but contained few files. This in term seemed to kill cifs and the web interface, though nfs was fine.

Long story short, it’s probably snmpd, there’s a bug in a version of the isilon os (possibly fixed now).

You can use fstat to find abnormally large open files (unfortunately lsof isn’t present, so I couldn’t see a way to locate unlinked files) and the process that has them open. You can then kill -9 snmpd. After that you can restart services as follows:

isi services apache2 disable
Isi services apache2 enable
isi services cifs disable
isi services cifs enable

You may also need to kill off webui/smbd (killall -9 isi_webui_d).

For blog updates and more follow me on twitter.