Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #172841

Re: system gobbles disk space

From Tony van der Hoff <tony@vanderhoff.org>
Newsgroups linux.debian.user
Subject Re: system gobbles disk space
Date 2016-09-27 11:00 +0200
Message-ID <slWp3-7kB-11@gated-at.bofh.it> (permalink)
References <slAHU-29M-11@gated-at.bofh.it> <slBNE-2PA-5@gated-at.bofh.it> <slELv-4MV-11@gated-at.bofh.it> <slFex-4Zu-1@gated-at.bofh.it> <slFHz-5oB-1@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

On 26/09/16 17:04, tomas@tuxteam.de wrote:
> On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
>> On 26/09/16 16:03, tomas@tuxteam.de wrote:
>>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
>>>
>>>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
>>>

Thanks again to all who have helped with this.

After the overnight run, I'm now seeing this:

root@shell:~# lsof +L1
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
mysqld     3164    mysql    4u   REG 254,16        0     0  589839
/tmp/ibvh7MKT (deleted)
mysqld     3164    mysql    5u   REG 254,16      473     0  589840
/tmp/ibT7D133 (deleted)
mysqld     3164    mysql    6u   REG 254,16        0     0  589845
/tmp/ibO9pgne (deleted)
mysqld     3164    mysql    7u   REG 254,16        0     0  589851
/tmp/ibOgMl6y (deleted)
mysqld     3164    mysql   11u   REG 254,16        0     0  589852
/tmp/ibON9JEJ (deleted)
/usr/sbin  4643 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin  4643 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin  4643 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin  4643 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin  4643 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin  4643 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data   39w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 13920 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 13920 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 13920 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 13920 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   39w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data   40w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   33w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   34w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data   39w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   29w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16711 www-data   30w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16711 www-data   31w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16711 www-data   32w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16711 www-data   33w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   34w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16768 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16768 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16768 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16768 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   37w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data   38w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16770 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16770 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16770 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16770 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16798 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16798 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16798 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16798 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data   36w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data   30w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 18785 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 18785 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 18785 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 18785 www-data   35w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data    2w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data   29w   REG 254,16    22055     0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data   31w   REG 254,16        0     0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 18929 www-data   32w   REG 254,16        0     0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 18929 www-data   33w   REG 254,16        0     0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 18929 www-data   34w   REG 254,16    10374     0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)


So, I can see this growing to an impossible size, which is probably
where my problem lies.

These files are created by a apache-driven PERL application, and the log
files are renamed/compressed overnight by a cron job.

This setup has existed for a number of years unchanged, so what's caused
it to start misbehaving?

How to stop it happening?
Should I be stopping Apache while compressing the logs?

-- 
Tony van der Hoff  | mailto:tony@vanderhoff.org
Ariège, France     |

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-26 11:50 +0200
  Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 12:10 +0200
  Re: system gobbles disk space Erwan David <erwan@rail.eu.org> - 2016-09-26 13:00 +0200
    Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 16:10 +0200
      Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-26 16:40 +0200
        Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-26 17:10 +0200
          Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 11:00 +0200
            Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-27 13:20 +0200
              Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 16:50 +0200
                Re: system gobbles disk space Sven Hartge <sven@svenhartge.de> - 2016-09-27 18:00 +0200
                Re: system gobbles disk space <tomas@tuxteam.de> - 2016-09-27 18:40 +0200
                Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-27 18:40 +0200
  [SOLVED] Re: system gobbles disk space Tony van der Hoff <tony@vanderhoff.org> - 2016-09-29 11:10 +0200

csiph-web