Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #13558

Re: Pattern suggestion

Show all headers | View raw

FrenKy schrieb:
> Hi *,
> I have a huge file (~10GB) which I'm reading line by line. Each line has
> to be analyzed by many number of different analyzers. The problem I have
> is that to make it at least a bit performance optimized due to sometimes
> time consuming processing (usually because of delays due to external
> interfaces) i would need to make it heavily multithreaded.
> File should be read only once to reduce IO on disks.
>
> So I need "1 driver to many workers" pattern where workers are
> multithreaded.
>
> I have a solution now based on Observable/Observer that I use (and it
> works) but I'm not sure if it is the best way.
>
> Any suggestion would be appreciated.
> Thanks in advance!

Some penny of thoughts:

- Check whether the bottleneck is writing the
   result of the analysers and not reading the file,
   you might add extra workers and queses for writing
   the result.

- Check whether the analysers are all equally fast,
   otherwise the least performant analyser will
   delay the processing, even with some queues,
   since they will be limited in size.

Bye

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Pattern suggestion FrenKy <frenky__nn@gmail.com> - 2012-04-15 16:11 +0200
  Re: Pattern suggestion Rui Maciel <rui.maciel@gmail.com> - 2012-04-15 16:04 +0100
    Re: Pattern suggestion Lew <noone@lewscanon.com> - 2012-04-15 08:15 -0700
  Re: Pattern suggestion markspace <-@.> - 2012-04-15 08:17 -0700
    Re: Pattern suggestion Arne Vajhøj <arne@vajhoej.dk> - 2012-04-15 22:01 -0400
  Re: Pattern suggestion Jan Burse <janburse@fastmail.fm> - 2012-04-15 17:41 +0200
    Re: Pattern suggestion Jan Burse <janburse@fastmail.fm> - 2012-04-16 00:37 +0200
  Re: Pattern suggestion Patricia Shanahan <pats@acm.org> - 2012-04-15 09:17 -0700
    Re: Pattern suggestion Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-04-15 13:57 -0300
      Re: Pattern suggestion Martin Gregorie <martin@address-in-sig.invalid> - 2012-04-15 19:56 +0000
        Re: Pattern suggestion Robert Klemme <shortcutter@googlemail.com> - 2012-04-16 09:55 +0200
  Re: Pattern suggestion Arne Vajhøj <arne@vajhoej.dk> - 2012-04-15 21:58 -0400

csiph-web