Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #242383 > unrolled thread

what is flooding /var/tmp?

Back to article view | Back to linux.debian.user

Contents

  what is flooding /var/tmp? "sp007@caiway.net" <sp007@caiway.net> - 2021-11-25 04:40 +0100
    Re: what is flooding /var/tmp? Greg Wooledge <greg@wooledge.org> - 2021-11-25 05:00 +0100
      Re: what is flooding /var/tmp? "sp007@caiway.net" <sp007@caiway.net> - 2021-11-25 10:00 +0100
        Re: what is flooding /var/tmp? "sp007@caiway.net" <sp007@caiway.net> - 2021-11-26 01:40 +0100
    Re: what is flooding /var/tmp? Cindy Sue Causey <butterflybytes@gmail.com> - 2021-11-25 05:10 +0100
    Re: what is flooding /var/tmp? "sp007@caiway.net" <sp007@caiway.net> - 2021-11-25 20:40 +0100
    Re: what is flooding /var/tmp? Marc Auslander <marcausl@gmail.com> - 2021-11-25 22:50 +0100
      Re: what is flooding /var/tmp? Tim Woodall <debianuser@woodall.me.uk> - 2021-11-25 23:10 +0100

#242383 — what is flooding /var/tmp?

Hello,

My /var/tmp directory gets flooded by big files named:

sort01ei1t
sort01Eq7u
sort01sLAs
...
sortzZZtvv


the files are approx. 13 Gb each.
In 24 hours > 6000 are written.

My big partition is filled by it until the system freezes.

The files are plain text files, containing sshfs paths:

/mnt/nas/sshfs/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/
.........

nas and desktop are running debian 11 daily updated.

How can I find out which program is writing these files?

Thanks!

[toc] | [next] | [standalone]

#242384

On Thu, Nov 25, 2021 at 04:23:23AM +0100, sp007@caiway.net wrote:
> Hello,
> 
> My /var/tmp directory gets flooded by big files named:
> 
> sort01ei1t
> sort01Eq7u
> sort01sLAs
> ...
> sortzZZtvv
> 
> 
> the files are approx. 13 Gb each.
> In 24 hours > 6000 are written.

> How can I find out which program is writing these files?

They're almost certainly temporary files created by sort(1).

"strings /usr/bin/sort | grep sortX" pretty much confirms this.  The
program contains the string /sortXXXXXX which is a template that
could be used by mktemp(3) or a similar routine for temp file name
generation, and it matches the length of your filenames.

So, you're looking for a shell script or a shell command that uses sort,
and has a VERY large input.

Since the files appear on a regular basis, you're probably looking for
something triggered by cron, or a systemd timer.

If you can figure out when the files start appearing, you can narrow
down the start time of the job.  If it's the same start time every day,
that could help you identify which job it is, in case there are lots.
Then again, most systems don't have a huge number of cron jobs, so it
shouldn't be hard to look at all of them.

If it's a different time every day, then you might be looking for systemd
timers.  They're weird and I don't really understand them yet, so if you
have to tackle one of them, good luck.

[toc] | [prev] | [next] | [standalone]

#242389

On Wed, 24 Nov 2021 22:59:06 -0500
Greg Wooledge <greg@wooledge.org> wrote:

> On Thu, Nov 25, 2021 at 04:23:23AM +0100, sp007@caiway.net wrote:
> > Hello,
> > 
> > My /var/tmp directory gets flooded by big files named:
> > 
> > sort01ei1t
> > sort01Eq7u
> > sort01sLAs
> > ...
> > sortzZZtvv
> > 
> > 
> > the files are approx. 13 Gb each.
> > In 24 hours > 6000 are written.
> 
> > How can I find out which program is writing these files?
> 
> They're almost certainly temporary files created by sort(1).
> 
> "strings /usr/bin/sort | grep sortX" pretty much confirms this.  The
> program contains the string /sortXXXXXX which is a template that
> could be used by mktemp(3) or a similar routine for temp file name
> generation, and it matches the length of your filenames.
> 
> So, you're looking for a shell script or a shell command that uses
> sort, and has a VERY large input.
> 
> Since the files appear on a regular basis, you're probably looking for
> something triggered by cron, or a systemd timer.
> 
> If you can figure out when the files start appearing, you can narrow
> down the start time of the job.  If it's the same start time every
> day, that could help you identify which job it is, in case there are
> lots. Then again, most systems don't have a huge number of cron jobs,
> so it shouldn't be hard to look at all of them.
> 
> If it's a different time every day, then you might be looking for
> systemd timers.  They're weird and I don't really understand them
> yet, so if you have to tackle one of them, good luck.
> 

Thanks for your imput
[erhaps ot will find a clue

[toc] | [prev] | [next] | [standalone]

#242405

On Thu, 25 Nov 2021 09:53:21 +0100
"sp007@caiway.net" <sp007@caiway.net> wrote:

> On Wed, 24 Nov 2021 22:59:06 -0500
> Greg Wooledge <greg@wooledge.org> wrote:
> 
> > On Thu, Nov 25, 2021 at 04:23:23AM +0100, sp007@caiway.net wrote:
> > > Hello,
> > > 
> > > My /var/tmp directory gets flooded by big files named:
> > > 
> > > sort01ei1t
> > > sort01Eq7u
> > > sort01sLAs
> > > ...
> > > sortzZZtvv
> > > 

The files are generated by a cronjob for updatedb.

There is no file /etc/updatedb.conf so I created one, containing one
line:
PRUNEPATH="/mnt/nas"

When I run updatedb manually updatedb does also index /mnt/nas
So /etc/updatedb.conf doesn't work for locate

So I edited /etc/cron.daily/locate
The line 
PRUNEPATHS="/tmp /usr/tmp /var/tmp /afs /amd /alex /var/spool /sfs
/media /var/lib/schroot/mount" 
I changed to
PRUNEPATHS="/mnt/nas /tmp /usr/tmp /var/tmp /afs /amd /alex /var/spool
/sfs /media /var/lib/schroot/mount"

This should work I guess, as long as I do not run updatedb manually.


Then I installed mlocate, this has a /etc/updatdb.conf with a line in
it: PRUNEFS=".. fuse.sshfs .."

mlocate works out of the box
plocate also

locate has a severe bug with mounted sshfs

I switched to plocate


Thanks all

[toc] | [prev] | [next] | [standalone]

#242385

On 11/24/21, sp007@caiway.net <sp007@caiway.net> wrote:
> Hello,
>
> My /var/tmp directory gets flooded by big files named:
>
> sort01ei1t
> sort01Eq7u
> sort01sLAs
> ...
> sortzZZtvv
>
>
> the files are approx. 13 Gb each.
> In 24 hours > 6000 are written.
>
> My big partition is filled by it until the system freezes.
>
> The files are plain text files, containing sshfs paths:
>
> /mnt/nas/sshfs/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/
> .........
>
> nas and desktop are running debian 11 daily updated.
>
> How can I find out which program is writing these files?


Hi, and WOW. While you're waiting for others to chime in, I found the
following that references basically what you're saying AND
specifically about Debian:

https://serverfault.com/questions/281374/what-is-creating-var-tmp-sort-files/281380

Because I'm not versed in this, I'm not going to try to bring it over
here. I don't want to mess it up. That took at least six or eight
different ways of asking the same question before that potential
answer FINALLY popped up at the top of the search.

IMPORTANT: They mention that one might need to run that "lsof" on the
directory/files a few times if necessary before you finally receive
usable information. It apparently needs to catch the culprit in the
active act of doing whatever is going on.

Best wishes...

Cindy :)
-- 
Cindy-Sue Causey
Talking Rock, Pickens County, Georgia, USA
* runs with birdseed *

[toc] | [prev] | [next] | [standalone]

#242402

On Thu, 25 Nov 2021 04:23:23 +0100
"sp007@caiway.net" <sp007@caiway.net> wrote:

> Hello,
> 
> My /var/tmp directory gets flooded by big files named:
> 
> sort01ei1t
> sort01Eq7u
> sort01sLAs
> ...
> sortzZZtvv
> 
> 
> the files are approx. 13 Gb each.
> In 24 hours > 6000 are written.
> 

they are approx. 13 Mb each, correction

# lsof /var/tmp/sort*
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system
/run/user/1000/gvfs Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.

the files are owned by root




> My big partition is filled by it until the system freezes.
> 
> The files are plain text files, containing sshfs paths:
> 
> /mnt/nas/sshfs/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/
> .........
> 
> nas and desktop are running debian 11 daily updated.
> 
> How can I find out which program is writing these files?
> 
> Thanks!
> 
> 

[toc] | [prev] | [next] | [standalone]

#242403

On 11/24/2021 10:40 PM, sp007@caiway.net wrote:
> Hello,
> 
> My /var/tmp directory gets flooded by big files named:
> 
> sort01ei1t
> sort01Eq7u
> sort01sLAs
> ...
> sortzZZtvv
> 
> 
> the files are approx. 13 Gb each.
> In 24 hours > 6000 are written.
> 
> My big partition is filled by it until the system freezes.
> 
> The files are plain text files, containing sshfs paths:
> 
> /mnt/nas/sshfs/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/
> .........
> 
> nas and desktop are running debian 11 daily updated.
> 
> How can I find out which program is writing these files?
> 
> Thanks!
> 
You might look at file creation time and look in /var/log/syslog to see 
what CRON job is running when they are created.

[toc] | [prev] | [next] | [standalone]

#242404

On Thu, 25 Nov 2021, Marc Auslander wrote:

> On 11/24/2021 10:40 PM, sp007@caiway.net wrote:
>> Hello,
>> 
>> My /var/tmp directory gets flooded by big files named:
>> 
>> sort01ei1t
>> sort01Eq7u
>> sort01sLAs
>> ...
>> sortzZZtvv
>> 
>> 
>> the files are approx. 13 Gb each.
>> In 24 hours > 6000 are written.
>> 
>> My big partition is filled by it until the system freezes.
>> 
>> The files are plain text files, containing sshfs paths:
>> 
>> /mnt/nas/sshfs/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/self/task/413551/root/proc/
>> .........
>> 
>> nas and desktop are running debian 11 daily updated.
>> 
>> How can I find out which program is writing these files?
>> 
>> Thanks!
>> 
> You might look at file creation time and look in /var/log/syslog to see what 
> CRON job is running when they are created.
>
But >6000 in 24hrs suggests one every 10s which doesn't sound like cron.
Sounds more like something stuck in a loop and aborting/retrying.

To the OP, have you checked ps for any jobs that have been running for
unexpectedly long?

Is a reboot a possibility (or already tried)?

The content of the files looks odd too, something is following links on
a remote /proc. Is the file one huge line or many lines (of max path
length whatever that is, I'm not sure - 4k?)

Do you even need to export/mount the proc filesystem like this - is
there some way you can stop that happening?

I don't use it but locate, or some tool like that, might be trying to
build an index of /proc - which I'd assume would be avoided on a local
/proc but might need some config magic for a remote one.

If nothing else, is it possible to bind mount an empty directory over
/mnt/nas/sshfs/proc and see if that helps?

>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web