Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #16901

Re: How can you make idle processors pick up java work?

Show key headers only | View raw

Patricia Shanahan wrote:
> qwertmonkey wrote:
> ...
> 
>>   Please, let me know if you have a better idea of how 
>> to read very large text
>> files in a faster way (while using a regular/sub-optimal box) ~
> 
> ...

Another approach is to farm out one file per thread, rather 
than multiple threads per file.

Part of the game of concurrency is to figure out the right 
size for a chunk.

Your problem also seems amenable to a map-reduce approach.

> It sounds as though your throughput is limited by disk read head time -
> splitting across physical disks increases that resource.
> 
> You *may* be able to improve the efficiency of one drive by using NIO
> non-blocking reads to keep more disk prefetch reads going at once. Given
> a pool of reads, the drive can optimize their order to reduce total head
> movement, and therefore time the read head wastes not actually doing a
> transfer.

You can also perhaps improve efficiency with the one thread-per-file 
approach.

I/O in each thread should be bursty, especially if you use 
'BufferedReader' with large buffers. Each thread's I/O time 
is potentially CPU time for other threads. Also, multiple 
threads' I/O might bunch up for elevator seeking as Patricia 
describes.

However, threads hit a limit of usefulness if you get too many.
Fortunately that limit should be higher than that imposed by 
your I/O subsystem. You control this by setting your thread 
pool's maximum count and other control factors appropriately.

For the kind of situation you describe I'd predict the 
optimum maximum to be somewhere around two to four times your 
core count.

I don't know whether NIO or thread-based expansion will help 
you more. Even the predictions I am making are implicity 
heavily burdened with caveats to measure and take nothing for 
granted.

I'd likely use a one thread-per-file approach for structural 
reasons, without much initial concern for whether it's the 
best performing. It's a clean architecture that allows good 
locality of working variables and a natural path to a service 
approach. I wouldn't expect any optimization beyond large 
buffers to help individual file performance, so I'd plan for an 
architecture that scales well with increased hardware. One file 
thread per file scales well to one service call per file, 
readily distributed across a cluster. It also fits well with 
expansions to an I/O subsystem.

Amdahl's Law is your friend if you use it to plan units of 
work that have little to no common paths.

-- 
Lew

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

How can you make idle processors pick up java work? "qwertmonkey" <qwertmonkey@1:261/38.remove-x1c-this> - 2012-07-31 20:07 +0000
  Re: How can you make idle processors pick up java work? Patricia Shanahan <pats@acm.org> - 2012-07-31 22:07 -0700
    Re: How can you make idle processors pick up java work? Lew <lewbloch@gmail.com> - 2012-08-01 00:07 -0700

csiph-web