Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!npeer03.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!post01.iad.highwinds-media.com!newsfe04.iad.POSTED!8ad76e89!not-for-mail From: Arved Sandstrom User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.28) Gecko/20120313 Lightning/1.0b2 Thunderbird/3.1.20 MIME-Version: 1.0 Newsgroups: comp.lang.java.programmer Subject: Re: Pattern suggestion References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Lines: 44 Message-ID: X-Complaints-To: abuse@newsgroups-download.com NNTP-Posting-Date: Sun, 15 Apr 2012 16:57:43 UTC Organization: Public Usenet Newsgroup Access Date: Sun, 15 Apr 2012 13:57:42 -0300 X-Received-Bytes: 2862 Xref: csiph.com comp.lang.java.programmer:13561 On 12-04-15 01:17 PM, Patricia Shanahan wrote: > On 4/15/2012 7:11 AM, FrenKy wrote: >> Hi *, >> I have a huge file (~10GB) which I'm reading line by line. Each line has >> to be analyzed by many number of different analyzers. The problem I have >> is that to make it at least a bit performance optimized due to sometimes >> time consuming processing (usually because of delays due to external >> interfaces) i would need to make it heavily multithreaded. >> File should be read only once to reduce IO on disks. >> >> So I need "1 driver to many workers" pattern where workers are >> multithreaded. >> >> I have a solution now based on Observable/Observer that I use (and it >> works) but I'm not sure if it is the best way. > > I suggest taking a look at java.util.concurrent.ThreadPoolExecutor and > related classes. > > Try to minimize ordering relationships between processing on the lines, > so that you can overlap work on multiple lines as much as possible. > > Patricia I agree. A problem description like this, java.util.concurrent is the first thing that pops into my head. markspace mentioned map-reduce, and there is specifically fork-join in Java 1.7 (they are similar insofar as they are algorithms for dividing problems); I don't know if any of that's involved because the line analysis may be independent. IOW, this may not be a distributable problem, this may be millions of individual problems. java.util.concurrent will definitely have something. It could well be that the processing of each line is isolated, and I'd assuredly be thinking of ThreadPoolExecutor or something similar for managing these. It has a lot of tuning options including queues. If the analyzers for each line have to coordinate (and maybe there's some final processing after all complete) there are classes for that too, like CyclicBarrier. AHS -- A fly was very close to being called a "land," cause that's what they do half the time. -- Mitch Hedberg