Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #164249

Re: pipe a buffer to a program's stdin using popen

Show all headers | View raw

On 1/3/2022 9:07 AM, John Forkosh wrote:
> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>> John Forkosh <forkosh@panix.com> wrote:
>>> Kaz Kylheku <480-992-1380@kylheku.com> wrote:
>>>> On 2022-01-01, John Forkosh <forkosh@panix.com> wrote:
>>>>> Suppose I have a program that, in part, runs another program
>>>>> using FILE *outptr=popen("some command string","r") and then
>>>>> fread()'s its stdout from outptr. Now, if that other program
>>>>> is otherpgm which reads input from its stdin, and I also want
>>>>> that input to come from a file inputfile, then I could easily
>>>>> just write  FILE *outptr=popen("cat inputfile|otherpgm","r");
>>>>
>>>> That would be a useless use of cat; it could be done with:
>>>>    FILE *outptr=popen("otherpgm < inputfile","r");
>>>
>>> Thanks, Kaz:
>>> Yeah, I'm not actually doing either; the chosen illustration
>>> was basically just pipes versus redirection, and since I was
>>> kind of guessing the ultimate answer more likely involves pipes,
>>> I illustrated it along those lines.
>>>
>>>>> But here's the rub: rather than some inputfile on disk, my
>>>>> program has an internal  unsigned char buffer[9999]  which is
>>>>> what I want piped to otherpgm, but which I don't want written
>>>>> to disk first. How can I get buffer piped to otherpgm's stdin
>>>>> using popen()? Or some other mechanism, as long as I can read
>>>>> otherpgm's stdout from within my program.
>>>
>>>> A program to which you send input while reading its output
>>>> is a coprocess. The popen function is for simple uses; it doesn't
>>>> support coprocesses. You have to cob those together yourself
>>>> with fork, exec, and various other functions.
>>>
>>> Yeah, I've done a little, but not much, of that. And, as
>>> suggested in a preceding followup, I actually have a hardback
>>> paper copy of Stevens' APUE, which I occasionally refer to.
>>> But I couldn't quite figure out what to do from his Chapters
>>> 14 and 15, assuming that's where an answer's buried.
>>>
>>>> With a coprocess, there is the risk of deadlock. It happens
>>>> like this: you write output for the process to read, while
>>>> it concurrently reads it and processes output. Because you're
>>>> writing, you're not reading. The process blocks trying to
>>>> write the data to you, and therefore stops reading. Since it
>>>> stops reading, you block writing to it.
>>>
>>> Not a problem in my case. The 'otherpgm' reads all its input
>>> before it writes anything.
>>
>> That only exacerbates deadlock.
> 
> Oops, my bad. Guess I've got even more reading to do than I thought,
> before I can wrap my head around how to do this properly.
> 
>>>> In the general case, coprocesses require non-blocking I/O,
>>>> polling for input and output simultaneously, or else threads.
>>>> When the amount of data exchanged is small (so small that you can you
>>>> can rely on it all fitting into the pipe size on the given system)
>>>> deadlock may be avoided that way.
>>>>
>>>> In the case of just wanting to feed a canned piece of data to
>>>> the process from memory, while reading its output, we can use
>>>> fork() to create a process which will do the feeding.
>>>
>>> Feed how, exactly??? That had occurred to me (at least if I
>>
>> The feeding process (forked child) just sits in a loop sending the bytes
>> to one end of a pipe. That pipe is installed as the standard input
>> of the coprocess.
> 
> Yeah, the synopsis posted by Lew
>     >> pipe() to create a pair of unidirectional pipes
>     >> fork() to create a child process
>     >> in parent, close() the [0] pipe, and write() your data to the [1] pipe

I don't want to confuse things, but, while the above is correct (parent 
closes p[0] and writes to p[1]), the part below is not:
child should close p[1], and read from p[0]. If you need to read from 
stdin, then I believe the [0] pipe should be dup'd to stdin accordingly 
(I've not used dup myself too much).

See 'man pipe' for a clear description and example code of this part.

>     >> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your
>     >> target "some command string" executable
> seems like a good outline to guide my further APUE reading,
> maybe along with the example (also cited earlier) at
>     http://unixwiz.net/techtips/remap-pipe-fds.html
> 
>>> correctly understand what you're saying), but as far as I could
>>> tell that forked child is in exactly the same predicament as the
>>> parent was in my original question: how does it pipe (or redirect
>>> or whatever) a large unsigned char buffer[9999] (actually
>>> malloc/realloc'ed, often larger than 20MB) from its memory to
>>> the stdin of that 'otherpgm'?
>>
>> Your shell can do it:
>>
>>     cat_memory()
>>     {
>>        echo "string in shell's memory"
>>     }
>>
>>     upcased=$(cat_memory | tr '[a-z]' '[A-Z]')
>>
>>     # upcased is now "STRING IN SHELL'S MEMORY"
>>
>> The shell forks a process which calls the function (i.e. does not exec
>> anything external). And it forks a process for the "tr" command. As it's
>> doing that, it hooks up the pipes in all the right ways and so it goes.
>>
>> The shell itself goes into a loop reading the output of the pipe,
>> so that it can gather it into a string via the $(...) syntax, and then
>> assign that to a variable when it's done.
> 
> Thanks, Kaz, Kenny, Lew, Keith...
>    Sounded to me like straightforward functionality when I initially
> asked how to do it, and therefore supposed there'd be a pretty easy
> and straightforward way to implement it.  But now seems to be a bit
> harder than I expected. I'll nevertheless continue studying APUE and
> all the stuff you've suggested, but for the time being just continue
> using the already-coded-and-working (but inelegant) solution -- just
> write the first program's output to a /tmp/file and then popen() the
> second program with an "-f /tmp/file" to read that file (and finally
> rm it). But I'll eventually re-implement that with your suggestions.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-01 09:35 +0000
  Re: pipe a buffer to a program's stdin using popen Mark Bluemel <mark.bluemel@gmail.com> - 2022-01-01 03:22 -0800
    Re: pipe a buffer to a program's stdin using popen Meredith Montgomery <mmontgomery@levado.to> - 2022-01-01 11:21 -0300
    Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 09:34 +0000
  Re: pipe a buffer to a program's stdin using popen Kaz Kylheku <480-992-1380@kylheku.com> - 2022-01-01 19:51 +0000
    Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-01 21:00 +0000
      Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 09:12 +0000
    Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 09:30 +0000
      Re: pipe a buffer to a program's stdin using popen Kaz Kylheku <480-992-1380@kylheku.com> - 2022-01-03 00:03 +0000
        Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-03 08:07 +0000
          Re: pipe a buffer to a program's stdin using popen Manfred <noname@add.invalid> - 2022-01-03 22:54 +0100
  Re: pipe a buffer to a program's stdin using popen Keith Thompson <Keith.S.Thompson+u@gmail.com> - 2022-01-01 15:00 -0800
    Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-02 00:01 +0000
  Re: pipe a buffer to a program's stdin using popen Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2022-01-02 12:54 +0000
    Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-02 14:54 +0000
      Re: pipe a buffer to a program's stdin using popen Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2022-01-02 15:37 +0000
      Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 18:00 +0000
        Re: pipe a buffer to a program's stdin using popen gazelle@shell.xmission.com (Kenny McCormack) - 2022-01-02 18:03 +0000
    Re: pipe a buffer to a program's stdin using popen John Forkosh <forkosh@panix.com> - 2022-01-02 17:51 +0000

csiph-web