Re: Named pipes vs. Unix sockets

From	Phillip Frabott <nntp@fulltermprivacy.com>
Newsgroups	comp.os.linux.misc
Subject	Re: Named pipes vs. Unix sockets
Date	2024-11-22 17:06 -0500
Organization	A noiseless patient Spider
Message-ID	<vhqv8f$1br87$1@dont-email.me> (permalink)
References	(12 earlier) <lqaoknF8btnU2@mid.individual.net> <lqaq6cF8btnU3@mid.individual.net> <lqau7sF8btnU4@mid.individual.net> <vhq6ag$176f0$1@dont-email.me> <vhqqu0$1b2t7$6@dont-email.me>

Show all headers | View raw

On 11/22/2024 15:52, Lawrence D'Oliveiro wrote:
> On Fri, 22 Nov 2024 10:00:31 -0500, Phillip Frabott wrote:
> 
>> At the moment we have to do 4.6GB per 0.5sec to keep up, and
>> 7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
>> out as much performance from pipes as we could but there just isn't much
>> we can do anymore.
> 
> Sounds like a job for a shared memory section.

That only works if the entire dataset originates on, and results within, 
the same system. Unfortunately this is not the case so it wouldn't be 
possible. This is an intermediate system between the intake and final 
processing systems which are further down the line.

Unless you can recommend a way to do shared memory across different 
machines without extra hardware. Currently we ingest data from a 
(external) /dev (unbuffered data stream) into a process on the machine 
that buffers the data (currently though an unnamed pipe, which as I said 
before is in the process of being replaced with UNIX socket) then that 
data pipes to another process (again, being replaced with UNIX socket 
soon) which converts the data stream from the buffer to block data and 
then categorizes those blocks based on data types and depending on those 
data types sends it to one of 4 /dev devices that leave the machine. The 
bottleneck is not between the first and second process (buffer process 
and stream to block data processor) it's between the intake /dev to the 
intermediate and the intermediate to the 4 outbound /devs. These /dev 
are not networks so there is no TCP data. They are hard lined between 
the /dev machine and the intermediate computer and the intermediate 
computer to the 4 other /dev machines. So how would I share memory 
between /dev machines from the 80s-90s and the intermediate computer 
without using any network or TCP/UDP capabilities?

(If it was not apparent. these are manufacturing machines.)

-- 
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-11-18 01:20 -0500
  Re: Are We Back to the "Wars" Now ? D <nospam@example.net> - 2024-11-18 10:13 +0100
    Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-11-19 21:57 -0500
      Re: Are We Back to the "Wars" Now ? D <nospam@example.net> - 2024-11-20 10:01 +0100
  Re: Are We Back to the "Wars" Now ? Pancho <Pancho.Jones@proton.me> - 2024-11-18 09:27 +0000
  Re: Are We Back to the "Wars" Now ? Richard Kettlewell <invalid@invalid.invalid> - 2024-11-18 09:54 +0000
  Re: Are We Back to the "Wars" Now ? Rich <rich@example.invalid> - 2024-11-18 10:46 +0000
    Re: Are We Back to the "Wars" Now ? Phillip Frabott <nntp@fulltermprivacy.com> - 2024-11-18 09:45 -0500
      Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-20 00:23 +0000
        Re: Are We Back to the "Wars" Now ? Phillip Frabott <nntp@fulltermprivacy.com> - 2024-11-21 02:05 -0500
          Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-21 07:22 +0000
            Re: Are We Back to the "Wars" Now ? Phillip Frabott <nntp@fulltermprivacy.com> - 2024-11-21 10:12 -0500
              Re: Are We Back to the "Wars" Now ? Richard Kettlewell <invalid@invalid.invalid> - 2024-11-21 18:38 +0000
                Re: Are We Back to the "Wars" Now ? Phillip Frabott <nntp@fulltermprivacy.com> - 2024-11-21 21:55 -0500
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-22 03:12 +0000
                Re: Are We Back to the "Wars" Now ? vallor <vallor@cultnix.org> - 2024-11-22 06:09 +0000
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-22 06:37 +0000
                Re: Are We Back to the "Wars" Now ? vallor <vallor@cultnix.org> - 2024-11-22 07:02 +0000
                Named pipes vs. Unix sockets (was: Re: Are We Back to the "Wars" Now ?) vallor <vallor@cultnix.org> - 2024-11-22 07:29 +0000
                Re: Named pipes vs. Unix sockets vallor <vallor@cultnix.org> - 2024-11-22 08:38 +0000
                Re: Named pipes vs. Unix sockets Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-22 09:37 +0000
                Re: Named pipes vs. Unix sockets Phillip Frabott <nntp@fulltermprivacy.com> - 2024-11-22 10:00 -0500
                Re: Named pipes vs. Unix sockets Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-22 20:52 +0000
                Re: Named pipes vs. Unix sockets Phillip Frabott <nntp@fulltermprivacy.com> - 2024-11-22 17:06 -0500
                Re: Named pipes vs. Unix sockets Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-23 00:42 +0000
                Re: Named pipes vs. Unix sockets rlhamil@smart.net (Richard L. Hamilton) - 2024-12-03 05:23 +0000
                Re: Named pipes vs. Unix sockets Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-12-03 06:20 +0000
                Re: Are We Back to the "Wars" Now ? Rich <rich@example.invalid> - 2024-11-22 19:11 +0000
                Re: Are We Back to the "Wars" Now ? vallor <vallor@cultnix.org> - 2024-11-23 03:25 +0000
                Re: Are We Back to the "Wars" Now ? Rich <rich@example.invalid> - 2024-11-23 04:28 +0000
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-23 04:50 +0000
                Pipes v. FIFOs (was: Are We Back to the "Wars" Now ?) Geoff Clare <geoff@clare.See-My-Signature.invalid> - 2024-11-26 14:21 +0000
                Re: Are We Back to the "Wars" Now ? Richard Kettlewell <invalid@invalid.invalid> - 2024-11-22 10:04 +0000
              Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-21 21:56 +0000
                Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-11-22 01:44 -0500
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-22 06:49 +0000
                Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-11-22 23:44 -0500
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-23 05:19 +0000
                Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-11-23 01:27 -0500
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-23 07:41 +0000
                Re: Are We Back to the "Wars" Now ? "Carlos E.R." <robin_listas@es.invalid> - 2024-11-23 14:39 +0100
                Re: Are We Back to the "Wars" Now ? Fritz Wuehler <fritz@spamexpire-202411.rodent.frell.theremailer.net> - 2024-11-23 22:32 +0100
                Re: Are We Back to the "Wars" Now ? "Carlos E.R." <robin_listas@es.invalid> - 2024-11-24 00:11 +0100
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-23 22:04 +0000
                Re: Are We Back to the "Wars" Now ? "Carlos E.R." <robin_listas@es.invalid> - 2024-11-24 00:11 +0100
                Re: Are We Back to the "Wars" Now ? The Natural Philosopher <tnp@invalid.invalid> - 2024-11-23 11:12 +0000
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-23 21:25 +0000
                Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-11-23 19:42 -0500
                Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-24 02:03 +0000
                Re: Are We Back to the "Wars" Now ? Rich <rich@example.invalid> - 2024-11-24 14:25 +0000
                Re: Are We Back to the "Wars" Now ? Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2024-11-24 15:48 +0000
                Re: Are We Back to the "Wars" Now ? Rich <rich@example.invalid> - 2024-11-24 16:56 +0000
                Re: Are We Back to the "Wars" Now ? Richard Kettlewell <invalid@invalid.invalid> - 2024-11-24 19:14 +0000
                Re: Are We Back to the "Wars" Now ? "Carlos E.R." <robin_listas@es.invalid> - 2024-11-24 20:32 +0100
                Re: Are We Back to the "Wars" Now ? Rich <rich@example.invalid> - 2024-11-24 19:36 +0000
  Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-11-20 00:22 +0000
    Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-12-03 01:19 -0500
      Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-12-03 06:49 +0000
        Re: Are We Back to the "Wars" Now ? "186282@ud0s4.net" <186283@ud0s4.net> - 2024-12-04 01:10 -0500
          Re: Are We Back to the "Wars" Now ? Lawrence D'Oliveiro <ldo@nz.invalid> - 2024-12-04 06:45 +0000

csiph-web