Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #172791

Re: system gobbles disk space

Show all headers | View raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Sep 26, 2016 at 11:42:41AM +0200, Tony van der Hoff wrote:
> Hi,
> 
> For the last few days, my jessie VM has apparently run out of disk space
> overnight. df shows 100% usage on / but du does not show any enormous files.
> 
> If I reboot, the disk usage goes back to 50% on a 40GB volume.
> 
> How can I determine what is eating the space?

Hm. Tough question. One traditional culprit for that kind of behaviour is
a file which has been deleted (thus having no entry in a directory in the
file system) but which is kept open by some process. One way to find that
would be to look into the /proc file system. Each process there (represented
by a directory named after the PID, the process ID) has a subdirectory called
'fd' containing a symlink to each file the process has open. Look for dangling
symlinks and ask yourself in each case whether the file behind this symlink
migh be responsible for the missing space.

A practical example:

First, I create a process which keeps a file open, writing into it from
time to time:

 | tomas@rasputin:~$ ( while true ; do echo "bip"; sleep 5 ; done ) > /tmp/logfile &
 | [1] 3642

This process has the PID 3642. In /proc/3642/fd, I find an entry for this
file:

 | tomas@rasputin:~$ ls -l /proc/3642/fd
 | total 0
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 0 -> /dev/pts/3
 | l-wx------ 1 tomas tomas 64 Sep 26 11:53 1 -> /tmp/logfile
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 2 -> /dev/pts/3
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 255 -> /dev/pts/3

Note that this is linked from the file named '1', which is the process's
standard output, as one would expect from the shell jargon '>' (redirect
stdout).

Now I get nasty and remove /tmp/logfile. Note that this doesn't remove
the file itself -- just that one directory entry (the file might have
more than one: it gets removed whenever nobody needs it). The process
continues running as if nothing happened. The file keeps growing (albeit
slowly in my case) and using up space:

 | tomas@rasputin:~$ rm /tmp/logfile
 | tomas@rasputin:~$ ls -l /proc/3642/fd
 | total 0
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 0 -> /dev/pts/3
 | l-wx------ 1 tomas tomas 64 Sep 26 11:53 1 -> /tmp/logfile (deleted)
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 2 -> /dev/pts/3
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 255 -> /dev/pts/3

See? Link -> is now marked (deleted). In my version of ls and console
it's printed now in a happy red colour. I can end the nonsense by
killing the process:

 | tomas@rasputin:~$ fg
 | ( while true; do
 |     echo "bip"; sleep 5;
 | done ) > /tmp/logfile
 | ^C

Of course, you'll have to become root to be able to move around
other process's entrails this way. In my small example I just
created the process, that made it easy.

There are other ways of making "disk space disappear", among others
mounting something over a non-empty directory.

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfo8zsACgkQBcgs9XrR2kakRgCfT0BNO3HAjTdYqb17vQXU6o1e
RdQAnimGR73J4lmXH3/4rtYoXDvrkc7u
=n1gG
-----END PGP SIGNATURE-----

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-26 11:50 +0200
  Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 12:10 +0200
  Re: system gobbles disk space Erwan David <erwan@rail.eu.org> - 2016-09-26 13:00 +0200
    Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 16:10 +0200
      Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-26 16:40 +0200
        Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 17:10 +0200
          Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 11:00 +0200
            Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-27 13:20 +0200
              Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 16:50 +0200
                Re: system gobbles disk space Sven Hartge <sven@svenhartge.de> - 2016-09-27 18:00 +0200
                Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-27 18:40 +0200
                Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 18:40 +0200
  [SOLVED] Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-29 11:10 +0200

csiph-web