Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!news-out.readnews.com!transit3.readnews.com!news-out.news.tds.net!newsreading01.news.tds.net!53ab2750!not-for-mail
From: "Patricia Shanahan" <patricia.shanahan@1:261/38.remove-x1c-this>
Subject: Re: How can you make idle processors pick up java work?
Message-ID: <50182C81.55847.calajapr@time.synchro.net>
X-Comment-To: qwertmonkey
Newsgroups: comp.lang.java.programmer
In-Reply-To: <50182C81.55845.calajapr@time.synchro.net>
References: <50182C81.55845.calajapr@time.synchro.net>
X-FTN-AREA: COMP.LANG.JAVA.PROGRAMMER
X-FTN-MSGID: 1:261/38 cc9bd899
X-FTN-REPLY: 1:261/38 98af8745
Content-Type: text/plain; charset=IBM437
Content-Transfer-Encoding: 8bit
X-Gateway: time.synchro.net [Synchronet 3.16a-Win32 NewsLink 1.98]
Lines: 37
Date: Tue, 31 Jul 2012 20:07:06 GMT
NNTP-Posting-Host: 69.21.70.65
X-Complaints-To: news@tds.net
X-Trace: newsreading01.news.tds.net 1343765226 69.21.70.65 (Tue, 31 Jul 2012 15:07:06 CDT)
NNTP-Posting-Date: Tue, 31 Jul 2012 15:07:06 CDT
Organization: tds.net
Xref: csiph.com comp.lang.java.programmer:16879

  To: qwertmonkey
From: "Patricia Shanahan" <patricia.shanahan@1:261/38.remove-dpk-this>

  To: qwertmonkey
From: Patricia Shanahan <pats@acm.org>

On 7/30/2012 4:27 PM, qwertmonkey@syberiaoutpost.ru wrote:
>>> Is there a way to make these processors pick up/share work also, or
>>> do you have to use some sort of scheduling framework on top of java?
>
>> Use multiple threads?
> ~
>   a) I need to actually scan large text files (10+ million lines).
>   b) On each line there is a NL sentence.
>   c) That processing should be run only once, but as fast as possible.
> ~
>   d) If you go:
>   d.1) int iPrx = Runtime.getRuntime().availableProcessors();
>   d.2) count all lines
>   d.3) split the file in (total lines)/iPrx
>   d.4) then run iPrx threads (or executable instances using a batch script)
>   the time you waste on d.2) and d.3) will make all that strat senseless

Why worry about splitting by actual line count, rather than by byte position in 
file?

Patricia

-+- BBBS/Li6 v4.10 Dada-1
 + Origin: Prism bbs (1:261/38)
-+- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24

--- BBBS/Li6 v4.10 Dada-1
 * Origin: Prism bbs (1:261/38)
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24