Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #16934

Re: How can you make idle processors pick up java work?

Show key headers only | View raw

  To: Patricia Shanahan
From: Lew <lewbloch@gmail.com>

Patricia Shanahan wrote:
> qwertmonkey wrote:
> ...
>
>>   Please, let me know if you have a better idea of how
>> to read very large text
>> files in a faster way (while using a regular/sub-optimal box) ~
>
> ...

Another approach is to farm out one file per thread, rather than multiple 
threads per file.

Part of the game of concurrency is to figure out the right size for a chunk.

Your problem also seems amenable to a map-reduce approach.

> It sounds as though your throughput is limited by disk read head time -
> splitting across physical disks increases that resource.
>
> You *may* be able to improve the efficiency of one drive by using NIO
> non-blocking reads to keep more disk prefetch reads going at once. Given
> a pool of reads, the drive can optimize their order to reduce total head
> movement, and therefore time the read head wastes not actually doing a
> transfer.

You can also perhaps improve efficiency with the one thread-per-file approach.

I/O in each thread should be bursty, especially if you use
'BufferedReader' with large buffers. Each thread's I/O time
is potentially CPU time for other threads. Also, multiple threads' I/O might 
bunch up for elevator seeking as Patricia describes.

However, threads hit a limit of usefulness if you get too many. Fortunately 
that limit should be higher than that imposed by your I/O subsystem. You 
control this by setting your thread pool's maximum count and other control 
factors appropriately.

For the kind of situation you describe I'd predict the optimum maximum to be 
somewhere around two to four times your core count.

I don't know whether NIO or thread-based expansion will help you more. Even the 
predictions I am making are implicity heavily burdened with caveats to measure 
and take nothing for granted.

I'd likely use a one thread-per-file approach for structural reasons, without 
much initial concern for whether it's the best performing. It's a clean 
architecture that allows good locality of working variables and a natural path 
to a service approach. I wouldn't expect any optimization beyond large buffers 
to help individual file performance, so I'd plan for an architecture that 
scales well with increased hardware. One file thread per file scales well to 
one service call per file, readily distributed across a cluster. It also fits 
well with expansions to an I/O subsystem.

Amdahl's Law is your friend if you use it to plan units of work that have 
little to no common paths.

--
Lew

--- BBBS/Li6 v4.10 Dada-1
 * Origin: Prism bbs (1:261/38)
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: How can you make idle processors pick up java work? "Patricia Shanahan" <patricia.shanahan@1:261/38.remove-s5i-this> - 2012-08-01 18:09 +0000
  Re: How can you make idle processors pick up java work? "Lew" <lew@1:261/38.remove-s5i-this> - 2012-08-01 18:09 +0000

csiph-web