Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!news-out.readnews.com!transit4.readnews.com!news-out.news.tds.net!newsreading01.news.tds.net!53ab2750!not-for-mail From: "Joshua Cranmer" Subject: Re: How can you make idle processors pick up java work? Message-ID: <50182C81.55848.calajapr@time.synchro.net> X-Comment-To: qwertmonkey Newsgroups: comp.lang.java.programmer In-Reply-To: <50182C81.55845.calajapr@time.synchro.net> References: <50182C81.55845.calajapr@time.synchro.net> X-FTN-AREA: COMP.LANG.JAVA.PROGRAMMER X-FTN-MSGID: 1:261/38 feb1db93 X-FTN-REPLY: 1:261/38 98af8745 Content-Type: text/plain; charset=IBM437 Content-Transfer-Encoding: 8bit X-Gateway: time.synchro.net [Synchronet 3.16a-Win32 NewsLink 1.98] Lines: 43 Date: Tue, 31 Jul 2012 20:07:07 GMT NNTP-Posting-Host: 69.21.70.65 X-Complaints-To: news@tds.net X-Trace: newsreading01.news.tds.net 1343765227 69.21.70.65 (Tue, 31 Jul 2012 15:07:07 CDT) NNTP-Posting-Date: Tue, 31 Jul 2012 15:07:07 CDT Organization: tds.net Xref: csiph.com comp.lang.java.programmer:16880 To: qwertmonkey From: "Joshua Cranmer" To: qwertmonkey From: Joshua Cranmer [Gah, your newsreader is incapable of threading posts correctly. Please find a non-broken one.] On 7/30/2012 7:27 PM, qwertmonkey@syberiaoutpost.ru wrote: >>> Is there a way to make these processors pick up/share work also, or >>> do you have to use some sort of scheduling framework on top of java? > >> Use multiple threads? > ~ > a) I need to actually scan large text files (10+ million lines). > b) On each line there is a NL sentence. > c) That processing should be run only once, but as fast as possible. Only 10M-line files? The easiest way to do this is to just make a ThreadPoolExecutor and have your main thread dispatch requests as fast as possible to the pool. Or you can do the work pooling yourself, which may be faster since you're not continually posting Runnable's, but timing results would be necessary to convince me. There are other options, but chances are, your disk drive is going to saturate first (in short, it involves reading non-consecutive pages of the file, which is generally a recipe for disaster). -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth -+- BBBS/Li6 v4.10 Dada-1 + Origin: Prism bbs (1:261/38) -+- Synchronet 3.16a-Win32 NewsLink 1.98 Time Warp of the Future BBS - telnet://time.synchro.net:24 --- BBBS/Li6 v4.10 Dada-1 * Origin: Prism bbs (1:261/38) --- Synchronet 3.16a-Win32 NewsLink 1.98 Time Warp of the Future BBS - telnet://time.synchro.net:24