Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #172790 > unrolled thread

system gobbles disk space

Started byTony van der Hoff <tony@vanderhoff.org>
First post2016-09-26 11:50 +0200
Last post2016-09-29 11:10 +0200
Articles 13 — 4 participants

Back to article view | Back to linux.debian.user

Contents

  system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-26 11:50 +0200
    Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 12:10 +0200
    Re: system gobbles disk space Erwan David <erwan@rail.eu.org> - 2016-09-26 13:00 +0200
      Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 16:10 +0200
        Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-26 16:40 +0200
          Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 17:10 +0200
            Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 11:00 +0200
              Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-27 13:20 +0200
                Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 16:50 +0200
                  Re: system gobbles disk space Sven Hartge <sven@svenhartge.de> - 2016-09-27 18:00 +0200
                    Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-27 18:40 +0200
                    Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 18:40 +0200
    [SOLVED] Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-29 11:10 +0200

#172790 — system gobbles disk space

FromTony van der Hoff <tony@vanderhoff.org>
Date2016-09-26 11:50 +0200
Subjectsystem gobbles disk space
Message-ID<slAHU-29M-11@gated-at.bofh.it>
Hi,

For the last few days, my jessie VM has apparently run out of disk space
overnight. df shows 100% usage on / but du does not show any enormous files.

If I reboot, the disk usage goes back to 50% on a 40GB volume.

How can I determine what is eating the space?

-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

[toc] | [next] | [standalone]

#172791

From<tomas@tuxteam.de>
Date2016-09-26 12:10 +0200
Message-ID<slB1f-2w1-11@gated-at.bofh.it>
In reply to#172790
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Sep 26, 2016 at 11:42:41AM +0200, Tony van der Hoff wrote:
> Hi,
> 
> For the last few days, my jessie VM has apparently run out of disk space
> overnight. df shows 100% usage on / but du does not show any enormous files.
> 
> If I reboot, the disk usage goes back to 50% on a 40GB volume.
> 
> How can I determine what is eating the space?

Hm. Tough question. One traditional culprit for that kind of behaviour is
a file which has been deleted (thus having no entry in a directory in the
file system) but which is kept open by some process. One way to find that
would be to look into the /proc file system. Each process there (represented
by a directory named after the PID, the process ID) has a subdirectory called
'fd' containing a symlink to each file the process has open. Look for dangling
symlinks and ask yourself in each case whether the file behind this symlink
migh be responsible for the missing space.

A practical example:

First, I create a process which keeps a file open, writing into it from
time to time:

 | tomas@rasputin:~$ ( while true ; do echo "bip"; sleep 5 ; done ) > /tmp/logfile &
 | [1] 3642

This process has the PID 3642. In /proc/3642/fd, I find an entry for this
file:

 | tomas@rasputin:~$ ls -l /proc/3642/fd
 | total 0
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 0 -> /dev/pts/3
 | l-wx------ 1 tomas tomas 64 Sep 26 11:53 1 -> /tmp/logfile
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 2 -> /dev/pts/3
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 255 -> /dev/pts/3

Note that this is linked from the file named '1', which is the process's
standard output, as one would expect from the shell jargon '>' (redirect
stdout).

Now I get nasty and remove /tmp/logfile. Note that this doesn't remove
the file itself -- just that one directory entry (the file might have
more than one: it gets removed whenever nobody needs it). The process
continues running as if nothing happened. The file keeps growing (albeit
slowly in my case) and using up space:

 | tomas@rasputin:~$ rm /tmp/logfile
 | tomas@rasputin:~$ ls -l /proc/3642/fd
 | total 0
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 0 -> /dev/pts/3
 | l-wx------ 1 tomas tomas 64 Sep 26 11:53 1 -> /tmp/logfile (deleted)
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 2 -> /dev/pts/3
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 255 -> /dev/pts/3

See? Link -> is now marked (deleted). In my version of ls and console
it's printed now in a happy red colour. I can end the nonsense by
killing the process:

 | tomas@rasputin:~$ fg
 | ( while true; do
 |     echo "bip"; sleep 5;
 | done ) > /tmp/logfile
 | ^C

Of course, you'll have to become root to be able to move around
other process's entrails this way. In my small example I just
created the process, that made it easy.

There are other ways of making "disk space disappear", among others
mounting something over a non-empty directory.

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfo8zsACgkQBcgs9XrR2kakRgCfT0BNO3HAjTdYqb17vQXU6o1e
RdQAnimGR73J4lmXH3/4rtYoXDvrkc7u
=n1gG
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#172793

FromErwan David <erwan@rail.eu.org>
Date2016-09-26 13:00 +0200
Message-ID<slBNE-2PA-5@gated-at.bofh.it>
In reply to#172790
On Mon, Sep 26, 2016 at 11:42:41AM CEST, Tony van der Hoff <tony@vanderhoff.org> said:
> Hi,
> 
> For the last few days, my jessie VM has apparently run out of disk space
> overnight. df shows 100% usage on / but du does not show any enormous files.
> 
> If I reboot, the disk usage goes back to 50% on a 40GB volume.
> 
> How can I determine what is eating the space?

A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)

[toc] | [prev] | [next] | [standalone]

#172800

From<tomas@tuxteam.de>
Date2016-09-26 16:10 +0200
Message-ID<slELv-4MV-11@gated-at.bofh.it>
In reply to#172793
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:

> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)

Short and sweet. That's even better :-)

- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfpKrUACgkQBcgs9XrR2kaIOACcCKX4tkiK7ngX4yGpq6Wwt/eY
g3cAnicvstprAM3EkEASN+SAPeWKlUUA
=vm+A
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#172802

FromTony van der Hoff <tony@vanderhoff.org>
Date2016-09-26 16:40 +0200
Message-ID<slFex-4Zu-1@gated-at.bofh.it>
In reply to#172800
On 26/09/16 16:03, tomas@tuxteam.de wrote:
> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
> 
>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
> 
> Short and sweet. That's even better :-)
> 
That's great; thanks Tomas and Erwan (and others who replied).

So, I'm seeing this:

root@shell:~# lsof +L1
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
mysqld  3164 mysql    4u   REG 254,16        0     0 589839
/tmp/ibvh7MKT (deleted)
mysqld  3164 mysql    5u   REG 254,16        0     0 589840
/tmp/ibT7D133 (deleted)
mysqld  3164 mysql    6u   REG 254,16        0     0 589845
/tmp/ibO9pgne (deleted)
mysqld  3164 mysql    7u   REG 254,16        0     0 589851
/tmp/ibOgMl6y (deleted)
mysqld  3164 mysql   11u   REG 254,16        0     0 589852
/tmp/ibON9JEJ (deleted)

My interpretation is that mysql has 5 deleted files of 0 size open which
are each taking up an inode. ls /tmp is empty.

I guess, if there were many (how many?) such entries, the disk would
appear full, if it ran out of  inodes.

Is that correct?


-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

[toc] | [prev] | [next] | [standalone]

#172807

From<tomas@tuxteam.de>
Date2016-09-26 17:10 +0200
Message-ID<slFHz-5oB-1@gated-at.bofh.it>
In reply to#172802
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
> On 26/09/16 16:03, tomas@tuxteam.de wrote:
> > On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
> > 
> >> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
> > 
> > Short and sweet. That's even better :-)
> > 
> That's great; thanks Tomas and Erwan (and others who replied).
> 
> So, I'm seeing this:
> 
> root@shell:~# lsof +L1
> COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
> mysqld  3164 mysql    4u   REG 254,16        0     0 589839
> /tmp/ibvh7MKT (deleted)
> mysqld  3164 mysql    5u   REG 254,16        0     0 589840
> /tmp/ibT7D133 (deleted)
> mysqld  3164 mysql    6u   REG 254,16        0     0 589845
> /tmp/ibO9pgne (deleted)
> mysqld  3164 mysql    7u   REG 254,16        0     0 589851
> /tmp/ibOgMl6y (deleted)
> mysqld  3164 mysql   11u   REG 254,16        0     0 589852
> /tmp/ibON9JEJ (deleted)
> 
> My interpretation is that mysql has 5 deleted files of 0 size open which
> are each taking up an inode. ls /tmp is empty.

That's how I read that too.

> I guess, if there were many (how many?) such entries, the disk would
> appear full, if it ran out of  inodes.
> 
> Is that correct?

Yes -- but you'd need a lot of inodes. I'd expect the mysql process to
run into some limit on open files earlier. You can get an idea of the
state of your file system with "tune2fs -l" (I'm assuming ext2/3/4 here).

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfpORYACgkQBcgs9XrR2kZouQCfYP4XLmlzWH5Zes4Nmcj+MoDx
GOkAnRwwlzQD+QoxEuzNQdcZo5sKlzdH
=h5zW
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#172841

FromTony van der Hoff <tony@vanderhoff.org>
Date2016-09-27 11:00 +0200
Message-ID<slWp3-7kB-11@gated-at.bofh.it>
In reply to#172807
On 26/09/16 17:04, tomas@tuxteam.de wrote:
> On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
>> On 26/09/16 16:03, tomas@tuxteam.de wrote:
>>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
>>>
>>>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
>>>

Thanks again to all who have helped with this.

After the overnight run, I'm now seeing this:

root@shell:~# lsof +L1
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
mysqld     3164    mysql    4u   REG 254,16        0     0  589839
/tmp/ibvh7MKT (deleted)
mysqld     3164    mysql    5u   REG 254,16      473     0  589840
/tmp/ibT7D133 (deleted)
mysqld     3164    mysql    6u   REG 254,16        0     0  589845
/tmp/ibO9pgne (deleted)
mysqld     3164    mysql    7u   REG 254,16        0     0  589851
/tmp/ibOgMl6y (deleted)
mysqld     3164    mysql   11u   REG 254,16        0     0  589852
/tmp/ibON9JEJ (deleted)
/usr/sbin  4643 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin  4643 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin  4643 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin  4643 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin  4643 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   39w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 13920 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 13920 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 13920 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 13920 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   39w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   40w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   33w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   34w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   39w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   29w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16711 www-data   30w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16711 www-data   31w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16711 www-data   32w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16711 www-data   33w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   34w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16768 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16768 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16768 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16768 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16770 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16770 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16770 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16770 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16798 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16798 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16798 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16798 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 18785 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 18785 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 18785 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 18785 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 18929 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 18929 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 18929 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)


So, I can see this growing to an impossible size, which is probably
where my problem lies.

These files are created by a apache-driven PERL application, and the log
files are renamed/compressed overnight by a cron job.

This setup has existed for a number of years unchanged, so what's caused
it to start misbehaving?

How to stop it happening?
Should I be stopping Apache while compressing the logs?

-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

[toc] | [prev] | [next] | [standalone]

#172847

From<tomas@tuxteam.de>
Date2016-09-27 13:20 +0200
Message-ID<slYAx-qw-9@gated-at.bofh.it>
In reply to#172841
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Sep 27, 2016 at 10:55:52AM +0200, Tony van der Hoff wrote:
> On 26/09/16 17:04, tomas@tuxteam.de wrote:
> > On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
> >> On 26/09/16 16:03, tomas@tuxteam.de wrote:
> >>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
> >>>
> >>>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
> >>>
> 
> Thanks again to all who have helped with this.
> 
> After the overnight run, I'm now seeing this:

[...]

Bad Apache, bad :-)

> So, I can see this growing to an impossible size, which is probably
> where my problem lies.
> 
> These files are created by a apache-driven PERL application, and the log
> files are renamed/compressed overnight by a cron job.

Seems this "cron job" is doing logrotate's job (or I am misunderstanding).

> This setup has existed for a number of years unchanged, so what's caused
> it to start misbehaving?
> 
> How to stop it happening?
> Should I be stopping Apache while compressing the logs?

There is a way to tell long-running processes to close their log files
and then re-open them (typically sending a HUP or USR1 signal, this has
established itself as a kind of "standard protocol"). This is usually
part of the logrotate process. You have logrotate to take care of your
log files, have you?

This is an example off logrotate's man page:

       /var/log/messages {
           rotate 5
           weekly
           postrotate
               /usr/bin/killall -HUP syslogd
           endscript
       }

Meaning "take care of log messages in this-and-that way and *after* that
send a HUP signal to syslogd". You should have a similar entry for your
Apache in /etc/logrotate.d or thereabouts.

If your Apache's log files aren't managed by logrotate (someone *is*
moving them around, after all) then go find out what Apache needs to
release its grip on log files and arrange for that to happen after
they have been moved around.

If I read [1] correctly, the Apache folks recommend a graceful restart
of the server after log rotation.

[1] https://httpd.apache.org/docs/2.4/logs.html#rotation

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfqU20ACgkQBcgs9XrR2kYgiQCeM0oR8IdvvzWce7ey/0LXeTIY
zOMAniQRK/N43LW2sNCp34aiboJToVa8
=4ccB
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#172855

FromTony van der Hoff <tony@vanderhoff.org>
Date2016-09-27 16:50 +0200
Message-ID<sm1RM-2fQ-23@gated-at.bofh.it>
In reply to#172847
On 27/09/16 13:09, tomas@tuxteam.de wrote:
> On Tue, Sep 27, 2016 at 10:55:52AM +0200, Tony van der Hoff wrote:
>> On 26/09/16 17:04, tomas@tuxteam.de wrote:
>>> On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
>>>> On 26/09/16 16:03, tomas@tuxteam.de wrote:
>>>>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
>>>>>
>>>>>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
>>>>>
> 
>> Thanks again to all who have helped with this.
> 
>> After the overnight run, I'm now seeing this:
> 
> [...]
> 
> Bad Apache, bad :-)
>
> Seems this "cron job" is doing logrotate's job (or I am misunderstanding).
> 
Well, I guess you're correct. It never occurred to me to use logrotate
for this purpose. I imagined that was exclusively for the system
packages. Evidently not!

However, it wouldn't work in this case. Various other applications  rely
on the names of the compressed files being in a particular format, which
I can't easily achieve with logrotate. So, I'll stick with my original
script but add "apachectl graceful" before fiddling with the files. The
only snag seems to be that the script now has to run as root. Not a
terrible deal.

>> This setup has existed for a number of years unchanged, so what's caused
>> it to start misbehaving?
> 
I still don't understand what's changed; this server is running
oldstable, so should only receive security updates. I guess it's just
taken a long time to fill up the file space with deleted files. Ho hum!

Thanks for your help.

-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

[toc] | [prev] | [next] | [standalone]

#172861

FromSven Hartge <sven@svenhartge.de>
Date2016-09-27 18:00 +0200
Message-ID<sm2Xv-2S0-33@gated-at.bofh.it>
In reply to#172855
Tony van der Hoff <tony@vanderhoff.org> wrote:

> However, it wouldn't work in this case. Various other applications
> rely on the names of the compressed files being in a particular
> format, which I can't easily achieve with logrotate. So, I'll stick
> with my original script but add "apachectl graceful" before fiddling
> with the files.

You should add this _after_ fiddling with the files. Reloading apache
before renaming/rotating the log files will not stop them from being
reopened again.

You rename/rotate the files _first_ and *then* reload apache to have the
intended effect.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

[toc] | [prev] | [next] | [standalone]

#172863

From<tomas@tuxteam.de>
Date2016-09-27 18:40 +0200
Message-ID<sm3Ad-3oW-1@gated-at.bofh.it>
In reply to#172861
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Sep 27, 2016 at 05:53:05PM +0200, Sven Hartge wrote:

[...]

> You should add this _after_ fiddling with the files [...]

Exactly.

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfqnz4ACgkQBcgs9XrR2kYOnQCfWlsVClBgkSyDIwUuYTQhKnT1
mo8AnRlsP57NdbZ9MKI+qXosIjiPBjFi
=aVbS
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#172864

FromTony van der Hoff <tony@vanderhoff.org>
Date2016-09-27 18:40 +0200
Message-ID<sm3Ad-3oW-11@gated-at.bofh.it>
In reply to#172861
On 27/09/16 17:53, Sven Hartge wrote:
> Tony van der Hoff <tony@vanderhoff.org> wrote:
> 
>> However, it wouldn't work in this case. Various other applications
>> rely on the names of the compressed files being in a particular
>> format, which I can't easily achieve with logrotate. So, I'll stick
>> with my original script but add "apachectl graceful" before fiddling
>> with the files.
> 
> You should add this _after_ fiddling with the files. Reloading apache
> before renaming/rotating the log files will not stop them from being
> reopened again.
> 
> You rename/rotate the files _first_ and *then* reload apache to have the
> intended effect.
> 
> Grüße,
> Sven.
> 

Oops, Thanks!

-- 
Tony van der Hoff        | mailto:tony@vanderhoff.org
Buckinghamshire, England |

[toc] | [prev] | [next] | [standalone]

#172908 — [SOLVED] Re: system gobbles disk space

FromTony van der Hoff <tony@vanderhoff.org>
Date2016-09-29 11:10 +0200
Subject[SOLVED] Re: system gobbles disk space
Message-ID<smFvP-250-17@gated-at.bofh.it>
In reply to#172790
On 26/09/16 11:42, Tony van der Hoff wrote:
> Hi,
> 
> For the last few days, my jessie VM has apparently run out of disk space
> overnight. df shows 100% usage on / but du does not show any enormous files.
> 
> If I reboot, the disk usage goes back to 50% on a 40GB volume.

Pleased to say, that with various people's help, this problem now seems
to be solved.

Thanks, all!



-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web