Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #30306 > unrolled thread

A stream/pipe that fills only on demand...

Back to article view | Back to comp.lang.java.programmer

Contents

  A stream/pipe that fills only on demand... Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-11-07 09:34 +0000
    Re: A stream/pipe that fills only on demand... Steven Simpson <ss@domain.invalid> - 2014-11-07 10:00 +0000
      Re: A stream/pipe that fills only on demand... Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-11-07 15:14 +0000
    Re: A stream/pipe that fills only on demand... Martin Gregorie <martin@address-in-sig.invalid> - 2014-11-07 12:27 +0000
      Re: A stream/pipe that fills only on demand... Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-11-07 15:13 +0000
    Re: A stream/pipe that fills only on demand... Robert Klemme <shortcutter@googlemail.com> - 2014-11-08 11:20 +0100
    Re: A stream/pipe that fills only on demand... Nigel Wade <nmw@ion.le.ac.uk> - 2014-11-10 09:24 +0000
      Re: A stream/pipe that fills only on demand... Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-11-29 16:02 +0000

#30306 — A stream/pipe that fills only on demand...

Normally, when one has a pipe-stream, the pipe gets filled on
one side, till its full, and drained on the other side while
data is needed. If less data is needed, then the remaining data
in the pipe is discarded. (simplified; no nitpicking, please)

My needs are a bit different, though:  I need a pipe-stream
that recognizes reader's request for bytes, and tells the 
writer-end about the requested volume, and then the writer
will write just as much into the pipe as is needed by the 
reader. The reader just uses the normal API that InputStream
offers, but I need these calls to identify themselves to the
pipe-writer, which would use a (yet unspecified) special API,
so the writer knows how much to put into the pipe-stream.

Background is, that the items that the writer will put into
the pipe-stream are "precious" and must not be wasted.
Kind of like reading back a stream of serialized data, where
each de-serializing object must not consume more than its own
bytes, but the Object needing its data is on a different host
than the byte stream. That's just an analogy - don't stress it.

Probably it's not rocket science to program it myself, but if
it rings a bell in someone, I'd appreciate hints about existing
technical terms and/or implementations.
I certainly do NOT expect you to implement it for me, and that's
also why I didn't write up a bullet-proof specificatiion :-)

[toc] | [next] | [standalone]

#30307

On 07/11/14 09:34, Andreas Leitgeb wrote:
> My needs are a bit different, though:  I need a pipe-stream
> that recognizes reader's request for bytes, and tells the
> writer-end about the requested volume, and then the writer
> will write just as much into the pipe as is needed by the
> reader. The reader just uses the normal API that InputStream
> offers, but I need these calls to identify themselves to the
> pipe-writer, which would use a (yet unspecified) special API,
> so the writer knows how much to put into the pipe-stream.

Surely, that special API is InputStream itself...?

-- 
ss at comp dot lancs dot ac dot uk

[toc] | [prev] | [next] | [standalone]

#30310

Steven Simpson <ss@domain.invalid> wrote:
> On 07/11/14 09:34, Andreas Leitgeb wrote:
>> My needs are a bit different, though:  I need a pipe-stream
>> that recognizes reader's request for bytes, and tells the
>> writer-end about the requested volume, and then the writer
>> will write just as much into the pipe as is needed by the
>> reader. The reader just uses the normal API that InputStream
>> offers, but I need these calls to identify themselves to the
>> pipe-writer, which would use a (yet unspecified) special API,
>> so the writer knows how much to put into the pipe-stream.
> Surely, that special API is InputStream itself...?

Yep, after having my mind run in circles for a while, that's
just what it will likely boil down to.

Thanks.

[toc] | [prev] | [next] | [standalone]

#30308

On Fri, 07 Nov 2014 09:34:15 +0000, Andreas Leitgeb wrote:

> Normally, when one has a pipe-stream, the pipe gets filled on one side,
> till its full, and drained on the other side while data is needed. If
> less data is needed, then the remaining data in the pipe is discarded.
> (simplified; no nitpicking, please)
> 
> My needs are a bit different, though:  I need a pipe-stream that
> recognizes reader's request for bytes, and tells the writer-end about
> the requested volume, and then the writer will write just as much into
> the pipe as is needed by the reader. The reader just uses the normal API
> that InputStream offers, but I need these calls to identify themselves
> to the pipe-writer, which would use a (yet unspecified) special API, so
> the writer knows how much to put into the pipe-stream.
> 
> Background is, that the items that the writer will put into the
> pipe-stream are "precious" and must not be wasted.
> Kind of like reading back a stream of serialized data, where each
> de-serializing object must not consume more than its own bytes, but the
> Object needing its data is on a different host than the byte stream.
> That's just an analogy - don't stress it.
> 
> Probably it's not rocket science to program it myself, but if it rings a
> bell in someone, I'd appreciate hints about existing technical terms
> and/or implementations.
> I certainly do NOT expect you to implement it for me, and that's also
> why I didn't write up a bullet-proof specificatiion :-)

What are the objections to using sockets to handle what is obviously some 
sort of request/response pairing? Is the response particularly large?


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#30309

Martin Gregorie <martin@address-in-sig.invalid> wrote:
> On Fri, 07 Nov 2014 09:34:15 +0000, Andreas Leitgeb wrote:
>> My needs are a bit different, though:  I need a pipe-stream that
>> recognizes reader's request for bytes, and tells the writer-end about
>> the requested volume, and then the writer will write just as much into
>> the pipe as is needed by the reader. The reader just uses the normal API
>> that InputStream offers, but I need these calls to identify themselves
>> to the pipe-writer, which would use a (yet unspecified) special API, so
>> the writer knows how much to put into the pipe-stream.
> What are the objections to using sockets to handle what is obviously some 
> sort of request/response pairing? Is the response particularly large?

Originally, I believed that there would be more to my problem, possibly
requiring inter-thread communication between a writer thread and a reader
thread, but it seems that this approach wouldn't have worked out, anyway.

I'll end up implementing StreamReader and have its read(...) methods
directly obtain data from the "precious" source, which isn't a stream
itself.

If I later find out, that I need threads afterall, then by that time
I hope I'll have a better grasp of the actual threading situation.

Thanks for caring. :-)

[toc] | [prev] | [next] | [standalone]

#30311

On 07.11.2014 10:34, Andreas Leitgeb wrote:
> Normally, when one has a pipe-stream, the pipe gets filled on
> one side, till its full, and drained on the other side while
> data is needed. If less data is needed, then the remaining data
> in the pipe is discarded. (simplified; no nitpicking, please)

That is not an accurate description of how this "normally" works.  If 
more data is in the pipe (which is just a queue) than is read it just 
sits there and will be retrieved later.  Nothing is discarded.

Also, whether a pipe is filled until it is full depends on the ratio 
between reading and writing speeds: if a reader is faster than the 
writer chances are that the pipe (read "bounded queue") never fills up.

All that assumes concurrent reading and writing.  That's where pipes are 
usually used.

Cheers

	robert

[toc] | [prev] | [next] | [standalone]

#30327

On 07/11/14 09:34, Andreas Leitgeb wrote:
> My needs are a bit different, though:  I need a pipe-stream
> that recognizes reader's request for bytes, and tells the
> writer-end about the requested volume, and then the writer
> will write just as much into the pipe as is needed by the
> reader. The reader just uses the normal API that InputStream
> offers, but I need these calls to identify themselves to the
> pipe-writer, which would use a (yet unspecified) special API,
> so the writer knows how much to put into the pipe-stream.

That's one scenario, but it's not the only one.
What you are essentially describing is a read request, write response.
The reader sends a request to the writer telling it what it wants and
the writer sends a response which the reader can read.

>
> Background is, that the items that the writer will put into
> the pipe-stream are "precious" and must not be wasted.
> Kind of like reading back a stream of serialized data, where
> each de-serializing object must not consume more than its own
> bytes, but the Object needing its data is on a different host
> than the byte stream. That's just an analogy - don't stress it.

When I need to do something similar I use either GenericServlet or RMI. Normally servlet for inter-host and RMI for 
inter-process, but there's no reason why you can't use either approach for either solution. Both of them are suitable 
for request/response operations. RMI has less overhead because it doesn't require a servlet engine, just the rmiregistry.

[toc] | [prev] | [next] | [standalone]

#30444

Nigel Wade <nmw@ion.le.ac.uk> wrote:
> On 07/11/14 09:34, Andreas Leitgeb wrote:
>> My needs are a bit different, though:  I need a pipe-stream
>> that recognizes reader's request for bytes, and tells the
>> writer-end about the requested volume, and then the writer
>> will write just as much into the pipe as is needed by the
>> reader. The reader just uses the normal API that InputStream
>> offers, but I need these calls to identify themselves to the
>> pipe-writer, which would use a (yet unspecified) special API,
>> so the writer knows how much to put into the pipe-stream.
>
> What you are essentially describing is a read request, write response.
> The reader sends a request to the writer telling it what it wants and
> the writer sends a response which the reader can read.

After thinking it through I end up agreeing. A pipe just isn't
the abstraction I needed for the task in the first place.

That cleared up, I wrote a custom InputStream subclass that 
implements the read methods to get the requested amount of 
data from some non-InputStream-based API and pass it back to
the caller.

Thanks for helping me organize my thoughts about it.

PS:
In case you've also skimmed my "jdbc setBinaryStream" thread: yes
I use the mechanism there as well, but that's not its primary use.
The challenge about "precious input" does obviously not apply to
the db-related use, as the stream is (well, I wish it were) read
to end, unlike for the use behind this thread.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web