Groups > comp.lang.java.programmer > #16787

Re: How can you make idle processors pick up java work?

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!news-out.readnews.com!transit4.readnews.com!news-out.news.tds.net!newsreading01.news.tds.net!53ab2750!not-for-mail
From	"Joshua Cranmer" <joshua.cranmer@1:261/38.remove-dpk-this>
Subject	Re: How can you make idle processors pick up java work?
Message-ID	<50181D64.55755.calajapr@time.synchro.net> (permalink)
X-Comment-To	qwertmonkey
Newsgroups	comp.lang.java.programmer
In-Reply-To	<50181D63.55752.calajapr@time.synchro.net>
References	<50181D63.55752.calajapr@time.synchro.net>
X-FTN-AREA	COMP.LANG.JAVA.PROGRAMMER
X-FTN-MSGID	1:261/38 1dcb6473
X-FTN-REPLY	1:261/38 310e48c1
Content-Type	text/plain; charset=IBM437
Content-Transfer-Encoding	8bit
X-Gateway	time.synchro.net [Synchronet 3.16a-Win32 NewsLink 1.98]
Lines	35
Date	Tue, 31 Jul 2012 18:02:37 GMT
NNTP-Posting-Host	69.21.70.65
X-Complaints-To	news@tds.net
X-Trace	newsreading01.news.tds.net 1343757757 69.21.70.65 (Tue, 31 Jul 2012 13:02:37 CDT)
NNTP-Posting-Date	Tue, 31 Jul 2012 13:02:37 CDT
Organization	tds.net
Xref	csiph.com comp.lang.java.programmer:16787

Show key headers only | View raw

  To: qwertmonkey
From: Joshua Cranmer <Pidgeot18@verizon.invalid>

[Gah, your newsreader is incapable of threading posts correctly. Please find a 
non-broken one.]

On 7/30/2012 7:27 PM, qwertmonkey@syberiaoutpost.ru wrote:
>>> Is there a way to make these processors pick up/share work also, or
>>> do you have to use some sort of scheduling framework on top of java?
>
>> Use multiple threads?
> ~
>   a) I need to actually scan large text files (10+ million lines).
>   b) On each line there is a NL sentence.
>   c) That processing should be run only once, but as fast as possible.

Only 10M-line files?

The easiest way to do this is to just make a ThreadPoolExecutor and have your 
main thread dispatch requests as fast as possible to the pool. Or you can do 
the work pooling yourself, which may be faster since you're not continually 
posting Runnable's, but timing results would be necessary to convince me.

There are other options, but chances are, your disk drive is going to saturate 
first (in short, it involves reading non-consecutive pages of the file, which 
is generally a recipe for disaster).

--
Beware of bugs in the above code; I have only proved it correct, not tried it. 
-- Donald E. Knuth

--- BBBS/Li6 v4.10 Dada-1
 * Origin: Prism bbs (1:261/38)
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24

Thread

How can you make idle processors pick up java work? "qwertmonkey" <qwertmonkey@1:261/38.remove-dpk-this> - 2012-07-31 18:02 +0000
  Re: How can you make idle processors pick up java work? "David Lamb" <david.lamb@1:261/38.remove-dpk-this> - 2012-07-31 18:02 +0000
  Re: How can you make idle processors pick up java work? "Patricia Shanahan" <patricia.shanahan@1:261/38.remove-dpk-this> - 2012-07-31 18:02 +0000
  Re: How can you make idle processors pick up java work? "Joshua Cranmer" <joshua.cranmer@1:261/38.remove-dpk-this> - 2012-07-31 18:02 +0000
  Re: How can you make idle processors pick up java work? "John B. Matthews" <john.b..matthews@1:261/38.remove-dpk-this> - 2012-07-31 18:02 +0000

csiph-web