Re: Noting EVERY File Write/Change/Delete in (near) Real Time ???

From	Robert Riches <spamtrap42@jacob21819.net>
Newsgroups	comp.os.linux.misc
Subject	Re: Noting EVERY File Write/Change/Delete in (near) Real Time ???
Date	2022-09-10 04:15 +0000
Organization	none-at-all
Message-ID	<slrntho3ra.c1o.spamtrap42@one.localnet> (permalink)
References	<QA-dna7ohPr3Nof-nZ2dnZfqn_th4p2d@earthlink.com> <slrnthlcod.ll0.spamtrap42@one.localnet> <ap6dncbHHsb6moH-nZ2dnZfqnPVj4p2d@earthlink.com>

Show all headers | View raw

On 2022-09-10, 25B.Z969 <25B.Z969@noda.net> wrote:
> On 9/8/22 11:29 PM, Robert Riches wrote:
>> On 2022-09-09, 25B.Z969 <25B.Z969@noda.net> wrote:
>>> There are a couple of monitoring functions in Linux, but
>>> they usually require being directed at specific folders,
>>> which is not practical if there are hundreds/thousands
>>> or worse.
>>>
>>> I look at softRAID ... and it's doing exactly what I'm
>>> looking for as it mirrors changes super-promptly. There
>>> is SOME tap into the lower-level kernel functions that
>>> shows everything going on as it happens. I just don't
>>> see a system function, or any GCC library, that can
>>> provide the same utility. I'm not interested in duplication
>>> though, just logging what's happened so I can do other
>>> stuff with the info.
>>>
>>> Yea, yea ... you can run rsync on everything over and
>>> over and over forever and have it build a log of what's
>>> changed. I've used that for other projects. Alas that's
>>> WAY too slow and uses too much CPU. Scanning is NOT
>>> the way, I want notice of EVERY file write/rewrite
>>> as it happens.
>>>
>>> Any ideas ? Words of wisdom ? SoftRAID tells me it CAN
>>> be done, but I can't penetrate the code well enough to
>>> see how it's doing the job.
>> 
>> If by "softRAID" you mean mdadm-style kernel-supported RAID, that
>> is for mirroring disks on a local machine.  It works really well
>> for that purpose.
>
>    Yes, it does. But at some level, in some way, it is
>    intercepting write/re-write/delete info on a per-file
>    basis so it can duplicate the action on the mirror
>    drive(s).

Nope.  The kernel-supported mdadm-style RAID works at the block
level, not at the file operation level.

>    I want that info, in (near) real time.
>
>> Perchance, might you be looking for something closer to a
>> Distributed Replicated Block Device, aka DRBD?
>> 
>> https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device
>
>    Um ... not exactly. I'm not looking to write to alternate
>    devices, I just need to know WHAT files are being written/deleted.
>
>    The OS knows this at some level - but, despite extensive searching,
>    I can't find it. I want to write a service that makes use of this
>    info, needs to be fairly quick, but a short queue of writes/deletes
>    that's not TOO deep is tolerable ... cope with them during 'slack'
>    CPU time. "Hello Dave, I just wrote "/A/B/C/D/E/F/G/junk.txt and
>    then deleted "/W/X/Y/Z/morejunk.txt".
>
>    Now there ARE a couple of system utilities, you generally
>    point them at a specific folder. I suspect they just keep
>    watching for datetime changes by scanning over and over
>    again quickly. I could write that in Python or 'C' during
>    lunch break. This is NOT efficient, NOT tenable for use
>    on "/" or multiple 12tb NAS drives - and you have to keep
>    potentially huge "previous state" tables.
>
>    Real-time as-its-done is what I'm looking for. Somehow I'm
>    sure it CAN be done, but I cannot find where to probe for
>    this info. Somwhere, just before drivers turn it into actual
>    disk sectors, is "I am committing to writing file
>    "/A/B/C/D/E/F/G/junk.txt".
>
>    This seems to be one of those places where Linux is as
>    opaque as Winders.

As opaque?  With Linux, the whole kernel source is there for you
to study.  A decade or so ago, I was assigned to write a device
mapper block device that would do intecept block-level
operations.  I studied several modules of kernel code to get an
idea of how to do that.  Fortunately, I was reassigned to a
different task before I had to start debugging that code.

-- 
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-08 22:55 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Robert Riches <spamtrap42@jacob21819.net> - 2022-09-09 03:29 +0000
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-09 23:41 -0400
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Robert Riches <spamtrap42@jacob21819.net> - 2022-09-10 04:15 +0000
        Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-10 22:51 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2022-09-09 01:19 -0400
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-09 08:21 +0100
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-09 23:56 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-09 08:22 +0100
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-10 00:09 -0400
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? The Natural Philosopher <tnp@invalid.invalid> - 2022-09-10 14:14 +0100
        Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-14 21:54 -0400
  Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Grant Taylor <gtaylor@tnetconsulting.net> - 2022-09-09 23:48 -0600
    Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? John-Paul Stewart <jpstewart@personalprojects.net> - 2022-09-12 19:48 -0400
      Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "25B.Z969" <25B.Z969@noda.net> - 2022-09-15 23:44 -0400
        Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-09-30 00:18 -0400
          Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Fritz Wuehler <fritz@spamexpire-202209.rodent.frell.theremailer.net> - 2022-09-30 14:39 +0200
            Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-01 00:30 -0400
              Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Joe Beanfish <joebeanfish@nospam.duh> - 2022-10-04 14:18 +0000
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Joe Beanfish <joebeanfish@nospam.duh> - 2022-10-04 14:25 +0000
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-14 23:30 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? The Natural Philosopher <tnp@invalid.invalid> - 2022-10-15 08:53 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-15 22:34 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? The Natural Philosopher <tnp@invalid.invalid> - 2022-10-16 11:12 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-16 21:51 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Anssi Saari <as@sci.fi> - 2022-10-17 11:50 +0300
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-10-17 17:58 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-10-16 12:24 +0100
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-16 21:12 -0400
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Rich <rich@example.invalid> - 2022-10-18 14:15 +0000
                Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? "26C.Z968" <26C.Z968@noada.net> - 2022-10-27 00:38 -0400
          Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Rich <rich@example.invalid> - 2022-09-30 14:18 +0000
            Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-30 20:54 +0100
          Re: Noting EVERY File Write/Change/Delete in (near) Real Time ??? Richard Kettlewell <invalid@invalid.invalid> - 2022-09-30 20:53 +0100

csiph-web