Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #164230

Re: pipe a buffer to a program's stdin using popen

Show key headers only | View raw

Kaz Kylheku <480-992-1380@kylheku.com> wrote:
> John Forkosh <forkosh@panix.com> wrote:
>> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>>> On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>>>> Suppose I have a program that, in part, runs another program
>>>> using FILE *outptr=popen("some command string","r") and then
>>>> fread()'s its stdout from outptr. Now, if that other program
>>>> is otherpgm which reads input from its stdin, and I also want
>>>> that input to come from a file inputfile, then I could easily
>>>> just write  FILE *outptr=popen("cat inputfile|otherpgm","r");
>>> 
>>> That would be a useless use of cat; it could be done with:
>>>   FILE *outptr=popen("otherpgm < inputfile","r");
>>
>> Thanks, Kaz:
>> Yeah, I'm not actually doing either; the chosen illustration
>> was basically just pipes versus redirection, and since I was
>> kind of guessing the ultimate answer more likely involves pipes,
>> I illustrated it along those lines.
>>
>>>> But here's the rub: rather than some inputfile on disk, my
>>>> program has an internal  unsigned char buffer[9999]  which is
>>>> what I want piped to otherpgm, but which I don't want written
>>>> to disk first. How can I get buffer piped to otherpgm's stdin
>>>> using popen()? Or some other mechanism, as long as I can read
>>>> otherpgm's stdout from within my program.
>>
>>> A program to which you send input while reading its output
>>> is a coprocess. The popen function is for simple uses; it doesn't
>>> support coprocesses. You have to cob those together yourself
>>> with fork, exec, and various other functions.
>>
>> Yeah, I've done a little, but not much, of that. And, as
>> suggested in a preceding followup, I actually have a hardback
>> paper copy of Stevens' APUE, which I occasionally refer to.
>> But I couldn't quite figure out what to do from his Chapters
>> 14 and 15, assuming that's where an answer's buried.
>>
>>> With a coprocess, there is the risk of deadlock. It happens
>>> like this: you write output for the process to read, while
>>> it concurrently reads it and processes output. Because you're
>>> writing, you're not reading. The process blocks trying to
>>> write the data to you, and therefore stops reading. Since it
>>> stops reading, you block writing to it.
>>
>> Not a problem in my case. The 'otherpgm' reads all its input
>> before it writes anything.
> 
> That only exacerbates deadlock.

Oops, my bad. Guess I've got even more reading to do than I thought,
before I can wrap my head around how to do this properly.

>>> In the general case, coprocesses require non-blocking I/O,
>>> polling for input and output simultaneously, or else threads.
>>> When the amount of data exchanged is small (so small that you can you
>>> can rely on it all fitting into the pipe size on the given system)
>>> deadlock may be avoided that way.
>>> 
>>> In the case of just wanting to feed a canned piece of data to
>>> the process from memory, while reading its output, we can use
>>> fork() to create a process which will do the feeding.
>>
>> Feed how, exactly??? That had occurred to me (at least if I
> 
> The feeding process (forked child) just sits in a loop sending the bytes
> to one end of a pipe. That pipe is installed as the standard input
> of the coprocess.

Yeah, the synopsis posted by Lew
   >> pipe() to create a pair of unidirectional pipes
   >> fork() to create a child process
   >> in parent, close() the [0] pipe, and write() your data to the [1] pipe
   >> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
   >> target "some command string" executable
seems like a good outline to guide my further APUE reading,
maybe along with the example (also cited earlier) at 
   http://unixwiz.net/techtips/remap-pipe-fds.html

>> correctly understand what you're saying), but as far as I could
>> tell that forked child is in exactly the same predicament as the
>> parent was in my original question: how does it pipe (or redirect
>> or whatever) a large unsigned char buffer[9999] (actually
>> malloc/realloc'ed, often larger than 20MB) from its memory to
>> the stdin of that 'otherpgm'?
> 
> Your shell can do it:
> 
>    cat_memory()
>    {
>       echo "string in shell's memory"
>    }
> 
>    upcased=$(cat_memory | tr '[a-z]' '[A-Z]')
> 
>    # upcased is now "STRING IN SHELL'S MEMORY"
> 
> The shell forks a process which calls the function (i.e. does not exec
> anything external). And it forks a process for the "tr" command. As it's
> doing that, it hooks up the pipes in all the right ways and so it goes.
> 
> The shell itself goes into a loop reading the output of the pipe,
> so that it can gather it into a string via the $(...) syntax, and then
> assign that to a variable when it's done.

Thanks, Kaz, Kenny, Lew, Keith...
  Sounded to me like straightforward functionality when I initially
asked how to do it, and therefore supposed there'd be a pretty easy
and straightforward way to implement it.  But now seems to be a bit
harder than I expected. I'll nevertheless continue studying APUE and
all the stuff you've suggested, but for the time being just continue
using the already-coded-and-working (but inelegant) solution -- just
write the first program's output to a /tmp/file and then popen() the
second program with an "-f /tmp/file" to read that file (and finally
rm it). But I'll eventually re-implement that with your suggestions.
-- 
John Forkosh  ( mailto:  j@f.com  where j=john and f=forkosh )

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-01 09:35 +0000
  Re: pipe a buffer to a program's stdin using popen Mark Bluemel <mark.bluemel@gmail.com> - 2022-01-01 03:22 -0800
    Re: pipe a buffer to a program's stdin using popen Meredith Montgomery <mmontgomery@levado.to> - 2022-01-01 11:21 -0300
    Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 09:34 +0000
  Re: pipe a buffer to a program's stdin using popen Kaz Kylheku <480-992-1380@kylheku.com> - 2022-01-01 19:51 +0000
    Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-01 21:00 +0000
      Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 09:12 +0000
    Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 09:30 +0000
      Re: pipe a buffer to a program's stdin using popen Kaz Kylheku <480-992-1380@kylheku.com> - 2022-01-03 00:03 +0000
        Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-03 08:07 +0000
          Re: pipe a buffer to a program's stdin using popen Manfred <noname@add.invalid> - 2022-01-03 22:54 +0100
  Re: pipe a buffer to a program's stdin using popen Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2022-01-01 15:00 -0800
    Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-02 00:01 +0000
  Re: pipe a buffer to a program's stdin using popen Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2022-01-02 12:54 +0000
    Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-02 14:54 +0000
      Re: pipe a buffer to a program's stdin using popen Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2022-01-02 15:37 +0000
      Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 18:00 +0000
        Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-02 18:03 +0000
    Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 17:51 +0000

csiph-web