Re: Noting EVERY File Write/Change/Delete in (near) Real Time ???

Subject	Re: Noting EVERY File Write/Change/Delete in (near) Real Time ???
Newsgroups	comp.os.linux.misc
References	<QA-dna7ohPr3Nof-nZ2dnZfqn_th4p2d@earthlink.com> <slrnthlcod.ll0.spamtrap42@one.localnet>
From	"25B.Z969" <25B.Z969@noda.net>
Date	2022-09-09 23:41 -0400
Message-ID	<ap6dncbHHsb6moH-nZ2dnZfqnPVj4p2d@earthlink.com> (permalink)

Show all headers | View raw

On 9/8/22 11:29 PM, Robert Riches wrote:
> On 2022-09-09, 25B.Z969 <25B.Z969@noda.net> wrote:
>> There are a couple of monitoring functions in Linux, but
>> they usually require being directed at specific folders,
>> which is not practical if there are hundreds/thousands
>> or worse.
>>
>> I look at softRAID ... and it's doing exactly what I'm
>> looking for as it mirrors changes super-promptly. There
>> is SOME tap into the lower-level kernel functions that
>> shows everything going on as it happens. I just don't
>> see a system function, or any GCC library, that can
>> provide the same utility. I'm not interested in duplication
>> though, just logging what's happened so I can do other
>> stuff with the info.
>>
>> Yea, yea ... you can run rsync on everything over and
>> over and over forever and have it build a log of what's
>> changed. I've used that for other projects. Alas that's
>> WAY too slow and uses too much CPU. Scanning is NOT
>> the way, I want notice of EVERY file write/rewrite
>> as it happens.
>>
>> Any ideas ? Words of wisdom ? SoftRAID tells me it CAN
>> be done, but I can't penetrate the code well enough to
>> see how it's doing the job.
> 
> If by "softRAID" you mean mdadm-style kernel-supported RAID, that
> is for mirroring disks on a local machine.  It works really well
> for that purpose.

   Yes, it does. But at some level, in some way, it is
   intercepting write/re-write/delete info on a per-file
   basis so it can duplicate the action on the mirror
   drive(s).

   I want that info, in (near) real time.

> Perchance, might you be looking for something closer to a
> Distributed Replicated Block Device, aka DRBD?
> 
> https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device

   Um ... not exactly. I'm not looking to write to alternate
   devices, I just need to know WHAT files are being written/deleted.

   The OS knows this at some level - but, despite extensive searching,
   I can't find it. I want to write a service that makes use of this
   info, needs to be fairly quick, but a short queue of writes/deletes
   that's not TOO deep is tolerable ... cope with them during 'slack'
   CPU time. "Hello Dave, I just wrote "/A/B/C/D/E/F/G/junk.txt and
   then deleted "/W/X/Y/Z/morejunk.txt".

   Now there ARE a couple of system utilities, you generally
   point them at a specific folder. I suspect they just keep
   watching for datetime changes by scanning over and over
   again quickly. I could write that in Python or 'C' during
   lunch break. This is NOT efficient, NOT tenable for use
   on "/" or multiple 12tb NAS drives - and you have to keep
   potentially huge "previous state" tables.

   Real-time as-its-done is what I'm looking for. Somehow I'm
   sure it CAN be done, but I cannot find where to probe for
   this info. Somwhere, just before drivers turn it into actual
   disk sectors, is "I am committing to writing file
   "/A/B/C/D/E/F/G/junk.txt".

   This seems to be one of those places where Linux is as
   opaque as Winders.

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-08 22:55 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Robert Riches <spamtrap42@jacob21819.net> - 2022-09-09 03:29 +0000
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-09 23:41 -0400
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Robert Riches <spamtrap42@jacob21819.net> - 2022-09-10 04:15 +0000
        Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-10 22:51 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2022-09-09 01:19 -0400
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-09 08:21 +0100
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-09 23:56 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-09 08:22 +0100
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-10 00:09 -0400
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? The Natural Philosopher <tnp@invalid.invalid> - 2022-09-10 14:14 +0100
        Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-14 21:54 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Grant Taylor <gtaylor@tnetconsulting.net> - 2022-09-09 23:48 -0600
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? John-Paul Stewart <jpstewart@personalprojects.net> - 2022-09-12 19:48 -0400
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-15 23:44 -0400
        Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-09-30 00:18 -0400
          Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Fritz Wuehler <fritz@spamexpire-202209.rodent.frell.theremailer.net> - 2022-09-30 14:39 +0200
            Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-01 00:30 -0400
              Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Joe Beanfish <joebeanfish@nospam.duh> - 2022-10-04 14:18 +0000
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Joe Beanfish <joebeanfish@nospam.duh> - 2022-10-04 14:25 +0000
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-14 23:30 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? The Natural Philosopher <tnp@invalid.invalid> - 2022-10-15 08:53 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-15 22:34 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? The Natural Philosopher <tnp@invalid.invalid> - 2022-10-16 11:12 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-16 21:51 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Anssi Saari <as@sci.fi> - 2022-10-17 11:50 +0300
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-10-17 17:58 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-10-16 12:24 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-16 21:12 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Rich <rich@example.invalid> - 2022-10-18 14:15 +0000
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-27 00:38 -0400
          Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Rich <rich@example.invalid> - 2022-09-30 14:18 +0000
            Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-30 20:54 +0100
          Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-30 20:53 +0100

csiph-web