Path: csiph.com!weretis.net!feeder6.news.weretis.net!panix!.POSTED.panix3.panix.com!not-for-mail From: John Forkosh Newsgroups: comp.lang.c Subject: Re: pipe a buffer to a program's stdin using popen Date: Mon, 3 Jan 2022 08:07:04 -0000 (UTC) Organization: PANIX Public Access Internet and UNIX, NYC Message-ID: References: <20220101101839.914@kylheku.com> <20220102155759.587@kylheku.com> Injection-Date: Mon, 3 Jan 2022 08:07:04 -0000 (UTC) Injection-Info: reader1.panix.com; posting-host="panix3.panix.com:166.84.1.3"; logging-data="1222"; mail-complaints-to="abuse@panix.com" User-Agent: tin/2.6.0-20210823 ("Coleburn") (NetBSD/9.2 (amd64)) Xref: csiph.com comp.lang.c:164230 Kaz Kylheku <480-992-1380@kylheku.com> wrote: > John Forkosh wrote: >> Kaz Kylheku <480-992-1380@kylheku.com> wrote: >>> On 2022-01-01, John Forkosh wrote: >>>> Suppose I have a program that, in part, runs another program >>>> using FILE *outptr=popen("some command string","r") and then >>>> fread()'s its stdout from outptr. Now, if that other program >>>> is otherpgm which reads input from its stdin, and I also want >>>> that input to come from a file inputfile, then I could easily >>>> just write FILE *outptr=popen("cat inputfile|otherpgm","r"); >>> >>> That would be a useless use of cat; it could be done with: >>> FILE *outptr=popen("otherpgm < inputfile","r"); >> >> Thanks, Kaz: >> Yeah, I'm not actually doing either; the chosen illustration >> was basically just pipes versus redirection, and since I was >> kind of guessing the ultimate answer more likely involves pipes, >> I illustrated it along those lines. >> >>>> But here's the rub: rather than some inputfile on disk, my >>>> program has an internal unsigned char buffer[9999] which is >>>> what I want piped to otherpgm, but which I don't want written >>>> to disk first. How can I get buffer piped to otherpgm's stdin >>>> using popen()? Or some other mechanism, as long as I can read >>>> otherpgm's stdout from within my program. >> >>> A program to which you send input while reading its output >>> is a coprocess. The popen function is for simple uses; it doesn't >>> support coprocesses. You have to cob those together yourself >>> with fork, exec, and various other functions. >> >> Yeah, I've done a little, but not much, of that. And, as >> suggested in a preceding followup, I actually have a hardback >> paper copy of Stevens' APUE, which I occasionally refer to. >> But I couldn't quite figure out what to do from his Chapters >> 14 and 15, assuming that's where an answer's buried. >> >>> With a coprocess, there is the risk of deadlock. It happens >>> like this: you write output for the process to read, while >>> it concurrently reads it and processes output. Because you're >>> writing, you're not reading. The process blocks trying to >>> write the data to you, and therefore stops reading. Since it >>> stops reading, you block writing to it. >> >> Not a problem in my case. The 'otherpgm' reads all its input >> before it writes anything. > > That only exacerbates deadlock. Oops, my bad. Guess I've got even more reading to do than I thought, before I can wrap my head around how to do this properly. >>> In the general case, coprocesses require non-blocking I/O, >>> polling for input and output simultaneously, or else threads. >>> When the amount of data exchanged is small (so small that you can you >>> can rely on it all fitting into the pipe size on the given system) >>> deadlock may be avoided that way. >>> >>> In the case of just wanting to feed a canned piece of data to >>> the process from memory, while reading its output, we can use >>> fork() to create a process which will do the feeding. >> >> Feed how, exactly??? That had occurred to me (at least if I > > The feeding process (forked child) just sits in a loop sending the bytes > to one end of a pipe. That pipe is installed as the standard input > of the coprocess. Yeah, the synopsis posted by Lew >> pipe() to create a pair of unidirectional pipes >> fork() to create a child process >> in parent, close() the [0] pipe, and write() your data to the [1] pipe >> in child, close(STDIN_FILENO), dup() the [1] pipe, and then exec() your >> target "some command string" executable seems like a good outline to guide my further APUE reading, maybe along with the example (also cited earlier) at http://unixwiz.net/techtips/remap-pipe-fds.html >> correctly understand what you're saying), but as far as I could >> tell that forked child is in exactly the same predicament as the >> parent was in my original question: how does it pipe (or redirect >> or whatever) a large unsigned char buffer[9999] (actually >> malloc/realloc'ed, often larger than 20MB) from its memory to >> the stdin of that 'otherpgm'? > > Your shell can do it: > > cat_memory() > { > echo "string in shell's memory" > } > > upcased=$(cat_memory | tr '[a-z]' '[A-Z]') > > # upcased is now "STRING IN SHELL'S MEMORY" > > The shell forks a process which calls the function (i.e. does not exec > anything external). And it forks a process for the "tr" command. As it's > doing that, it hooks up the pipes in all the right ways and so it goes. > > The shell itself goes into a loop reading the output of the pipe, > so that it can gather it into a string via the $(...) syntax, and then > assign that to a variable when it's done. Thanks, Kaz, Kenny, Lew, Keith... Sounded to me like straightforward functionality when I initially asked how to do it, and therefore supposed there'd be a pretty easy and straightforward way to implement it. But now seems to be a bit harder than I expected. I'll nevertheless continue studying APUE and all the stuff you've suggested, but for the time being just continue using the already-coded-and-working (but inelegant) solution -- just write the first program's output to a /tmp/file and then popen() the second program with an "-f /tmp/file" to read that file (and finally rm it). But I'll eventually re-implement that with your suggestions. -- John Forkosh ( mailto: j@f.com where j=john and f=forkosh )