Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.java.programmer > #13558
| Path | csiph.com!usenet.pasdenom.info!news.albasani.net!.POSTED!not-for-mail |
|---|---|
| From | Jan Burse <janburse@fastmail.fm> |
| Newsgroups | comp.lang.java.programmer |
| Subject | Re: Pattern suggestion |
| Date | Sun, 15 Apr 2012 17:41:25 +0200 |
| Organization | albasani.net |
| Lines | 31 |
| Message-ID | <jmeq76$f72$1@news.albasani.net> (permalink) |
| References | <jmel0t$jrh$1@news2.carnet.hr> |
| Mime-Version | 1.0 |
| Content-Type | text/plain; charset=ISO-8859-1; format=flowed |
| Content-Transfer-Encoding | 7bit |
| X-Trace | news.albasani.net 4JcGS5JJewiV85mLYGKrXfnFpS/ZYrNPIDNB6j/cHR0PCoCNZ5xYMcZueDPof2LoMKiKiAOaLtS1lHHqyEe2unMnGNTOFhlR+QYwDDhRV1h5k13wPHiJl7eTdQE/MvH6 |
| NNTP-Posting-Date | Sun, 15 Apr 2012 15:41:26 +0000 (UTC) |
| Injection-Info | news.albasani.net; logging-data="mwXcE9bdTV5jwEMvOt5yMif0aXmyFw7v5UTNcUUn1JpaSxQDzKFKAZddtll9RTWBRzUhhKU1BbEG3utCZxMduMvlNEulMO/WU0T0WKJ+QmtXMAq268/Eq1MbnUCE8Sf3"; mail-complaints-to="abuse@albasani.net" |
| User-Agent | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0) Gecko/20120312 Firefox/11.0 SeaMonkey/2.8 |
| In-Reply-To | <jmel0t$jrh$1@news2.carnet.hr> |
| Cancel-Lock | sha1:0vqt895plk36VTfZxq/Jhebxjpw= |
| Xref | csiph.com comp.lang.java.programmer:13558 |
Show key headers only | View raw
FrenKy schrieb: > Hi *, > I have a huge file (~10GB) which I'm reading line by line. Each line has > to be analyzed by many number of different analyzers. The problem I have > is that to make it at least a bit performance optimized due to sometimes > time consuming processing (usually because of delays due to external > interfaces) i would need to make it heavily multithreaded. > File should be read only once to reduce IO on disks. > > So I need "1 driver to many workers" pattern where workers are > multithreaded. > > I have a solution now based on Observable/Observer that I use (and it > works) but I'm not sure if it is the best way. > > Any suggestion would be appreciated. > Thanks in advance! Some penny of thoughts: - Check whether the bottleneck is writing the result of the analysers and not reading the file, you might add extra workers and queses for writing the result. - Check whether the analysers are all equally fast, otherwise the least performant analyser will delay the processing, even with some queues, since they will be limited in size. Bye
Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Pattern suggestion FrenKy <frenky__nn@gmail.com> - 2012-04-15 16:11 +0200
Re: Pattern suggestion Rui Maciel <rui.maciel@gmail.com> - 2012-04-15 16:04 +0100
Re: Pattern suggestion Lew <noone@lewscanon.com> - 2012-04-15 08:15 -0700
Re: Pattern suggestion markspace <-@.> - 2012-04-15 08:17 -0700
Re: Pattern suggestion Arne Vajhøj <arne@vajhoej.dk> - 2012-04-15 22:01 -0400
Re: Pattern suggestion Jan Burse <janburse@fastmail.fm> - 2012-04-15 17:41 +0200
Re: Pattern suggestion Jan Burse <janburse@fastmail.fm> - 2012-04-16 00:37 +0200
Re: Pattern suggestion Patricia Shanahan <pats@acm.org> - 2012-04-15 09:17 -0700
Re: Pattern suggestion Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2012-04-15 13:57 -0300
Re: Pattern suggestion Martin Gregorie <martin@address-in-sig.invalid> - 2012-04-15 19:56 +0000
Re: Pattern suggestion Robert Klemme <shortcutter@googlemail.com> - 2012-04-16 09:55 +0200
Re: Pattern suggestion Arne Vajhøj <arne@vajhoej.dk> - 2012-04-15 21:58 -0400
csiph-web