Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.development.system > #798

Re: File transfer for fastness

Show all headers | View raw

On Mon, 2020-06-15, Sathya Narayanan Thangavelu wrote:
> On Sunday, 14 June 2020 20:09:24 UTC+5:30, Jorgen Grahn  wrote:
>> On Sun, 2020-06-14, t.sathyanarayanan@gmail.com wrote:
>> > Hi,
>> >
>> > Im trying to find best solution for parallel file transfer, which is
>> > event driven after the file written to a directory with extension
>> > like .done, means the file is ready for file transfer.
>> >
>> > 1. Monitoring the directory for .done is cpu cycle time waster
>> > 2. running a cron on particular interval  to transfer files will
>> >    not help fasteness to the data.
>> 
>> I cannot understand what you want to do, or how you want to do it;
>> you would have to explain better.
>> 
>> > what are other options existing for  file transfer.
>> 
>> Personally I always use rsync over ssh.  Unless I download files over
>> http, then I use firefox or curl.  Bittorrent is another option.
...
>
> we have a distrubuted systems which collects peta bytes of data and
> deliver the data to netstorage, the problem we have is transmission
> of files at faster rate, the one problem we sees as soon as the file
> comes in, we are not delivering the data, we have to wait for cron
> to run and pick those files and to deliver netstorage, the problem
> in cron is we schedule a time to run but practically this will lead
> to big data backlog, my question here is there any event based for
> monitoring file system to initiate the job as soons as files that
> comes into the directory, like (epoll mechanism)

I think I see. Yes, that's the Inotify mechanism someone else wrote
about.

I think it would be better to avoid that (files appearing in a
directory as an IPC mechanism) but I can see how design decisions can
make it unavoidable.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .     .
\X/     snipabacken.se>   O  o   .

Back to comp.os.linux.development.system | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

File transfer for fastness t.sathyanarayanan@gmail.com - 2020-06-14 06:43 -0700
  Re: File transfer for fastness Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-06-14 14:39 +0000
    Re: File transfer for fastness Sathya Narayanan Thangavelu <t.sathyanarayanan@gmail.com> - 2020-06-15 02:14 -0700
      Re: File transfer for fastness Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-06-15 11:42 +0000
  Re: File transfer for fastness Grant Edwards <invalid@invalid.invalid> - 2020-06-15 01:31 +0000
    Re: File transfer for fastness Jorgen Grahn <grahn+nntp@snipabacken.se> - 2020-06-15 08:41 +0000

csiph-web