Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #13555

Re: Pattern suggestion

Show all headers | View raw

FrenKy wrote:

> Hi *,
> I have a huge file (~10GB) which I'm reading line by line. Each line has
> to be analyzed by many number of different analyzers. The problem I have
> is that to make it at least a bit performance optimized due to sometimes
> time consuming processing (usually because of delays due to external
> interfaces) i would need to make it heavily multithreaded.
> File should be read only once to reduce IO on disks.
> 
> So I need "1 driver to many workers" pattern where workers are
> multithreaded.
> 
> I have a solution now based on Observable/Observer that I use (and it
> works) but I'm not sure if it is the best way.
> 
> Any suggestion would be appreciated.
> Thanks in advance!

What about this:

Consider three types of component: the file reader, data analyzer and a data 
queue.

The data queue component acts as a data buffer.  This component is 
responsible for managing the data queue, and supplying the data analyzer 
components with the data from the file.  It buffers the data contained in a 
file by storing it in multiple data snippets in such a way that can be 
accessed concurrently by each data analyzer components.  Once a data snippet 
is analyzed by all data analyzers, the data queue frees the resource.

The file reader component is responsible for supplying the data queue 
component with data read from a file.  It accomplishes this by pushing data 
snippets on top of the data queue on request.

The data analyzer component requests data snippets from the data queue 
component.  Once the data is analyzed, the data analyzer component releases 
the resource and requests more data from the file by signaling the data 
queue for the next data snippet.

This scheme would, at least in theory, provide you with a way to have 
multiple data analyzers analyze independent parts of a file without forcing 
a set of data analyziers to wait for a component to finish processing a data 
snippet.  You can also avoid the observer pattern with this sort of 
solution, and therefore end up with a component which is simpler to 
understand and a bit more robust to change.


Hope this helps,
Rui Maciel

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Pattern suggestion FrenKy <frenky__nn@gmail.com> - 2012-04-15 16:11 +0200
  Re: Pattern suggestion Rui Maciel <rui.maciel@gmail.com> - 2012-04-15 16:04 +0100
    Re: Pattern suggestion Lew <noone@lewscanon.com> - 2012-04-15 08:15 -0700
  Re: Pattern suggestion markspace <-@.> - 2012-04-15 08:17 -0700
    Re: Pattern suggestion Arne Vajhøj <arne@vajhoej.dk> - 2012-04-15 22:01 -0400
  Re: Pattern suggestion Jan Burse <janburse@fastmail.fm> - 2012-04-15 17:41 +0200
    Re: Pattern suggestion Jan Burse <janburse@fastmail.fm> - 2012-04-16 00:37 +0200
  Re: Pattern suggestion Patricia Shanahan <pats@acm.org> - 2012-04-15 09:17 -0700
    Re: Pattern suggestion Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-04-15 13:57 -0300
      Re: Pattern suggestion Martin Gregorie <martin@address-in-sig.invalid> - 2012-04-15 19:56 +0000
        Re: Pattern suggestion Robert Klemme <shortcutter@googlemail.com> - 2012-04-16 09:55 +0200
  Re: Pattern suggestion Arne Vajhøj <arne@vajhoej.dk> - 2012-04-15 21:58 -0400

csiph-web