Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #16891 > unrolled thread

How can you make idle processors pick up java work?

Back to article view | Back to comp.lang.java.programmer

Contents

  How can you make idle processors pick up java work? "qwertmonkey" <qwertmonkey@1:261/38.remove-x1c-this> - 2012-07-31 20:07 +0000
    Re: How can you make idle processors pick up java work? Patricia Shanahan <pats@acm.org> - 2012-07-31 22:07 -0700
      Re: How can you make idle processors pick up java work? Lew <lewbloch@gmail.com> - 2012-08-01 00:07 -0700

#16891 — How can you make idle processors pick up java work?

From: "qwertmonkey" <qwertmonkey@1:261/38.remove-dpk-this>

From: qwertmonkey@syberianoutpost.ru

~
 Well, yes. I am not sure if I am right (or the extent to which ...),
but it seems I am dealing here with a physical barrier, that may not be 
friendly to "logical" ways. Unless the original file is split into and placed 
in 8 different physical disks, the data transfer does not find bottlenecks 
through the memory subsystem, ...
~
 While reading the characters by means of a (NIO 2 Files.new)BufferedReader
(which uses non-blocking I/O, right?) and parsing the sentences myself, I can 
read the whole file on a laptop based on an AMD Turion 64X2 Dual Core and 4Gb 
in less than 10 seconds
~
    BufferedReader BfR = Files.newBufferedReader(IFlPth,
Charset.forName("UTF-8"));
    char[] cBfr = new char[iBL];
// __
    int iRdByts = BfR.read(cBfr, iBfr00, iBfr02);
    while(iRdByts > -1){
     for(int i = iBfr00; (i < iRdByts); ++i){
      if(cBfr[i] == cLF){ ++lLns; }
     }
     iRdByts = BfR.read(cBfr, iBfr00, iBfr02);
    }// (iRdByts > -1)
// __
    BfR.close();
~
 I still have to code the logic on top of it, but that is much, much better
than it was taking before
~
 Please, let me know if you have a better idea of how to read very large text
files in a faster way (while using a regular/sub-optimal box) ~
 lbrtchx

-+- BBBS/Li6 v4.10 Dada-1
 + Origin: Prism bbs (1:261/38)
-+- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24

--- BBBS/Li6 v4.10 Dada-1
 * Origin: Prism bbs (1:261/38)
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24

[toc] | [next] | [standalone]

#16900

On 7/31/2012 1:07 PM, qwertmonkey wrote:
...
>   Please, let me know if you have a better idea of how to read very large text
> files in a faster way (while using a regular/sub-optimal box) ~
...

It sounds as though your throughput is limited by disk read head time -
splitting across physical disks increases that resource.

You *may* be able to improve the efficiency of one drive by using NIO
non-blocking reads to keep more disk prefetch reads going at once. Given
a pool of reads, the drive can optimize their order to reduce total head
movement, and therefore time the read head wastes not actually doing a
transfer.

Patricia

[toc] | [prev] | [next] | [standalone]

#16901

Patricia Shanahan wrote:
> qwertmonkey wrote:
> ...
> 
>>   Please, let me know if you have a better idea of how 
>> to read very large text
>> files in a faster way (while using a regular/sub-optimal box) ~
> 
> ...

Another approach is to farm out one file per thread, rather 
than multiple threads per file.

Part of the game of concurrency is to figure out the right 
size for a chunk.

Your problem also seems amenable to a map-reduce approach.

> It sounds as though your throughput is limited by disk read head time -
> splitting across physical disks increases that resource.
> 
> You *may* be able to improve the efficiency of one drive by using NIO
> non-blocking reads to keep more disk prefetch reads going at once. Given
> a pool of reads, the drive can optimize their order to reduce total head
> movement, and therefore time the read head wastes not actually doing a
> transfer.

You can also perhaps improve efficiency with the one thread-per-file 
approach.

I/O in each thread should be bursty, especially if you use 
'BufferedReader' with large buffers. Each thread's I/O time 
is potentially CPU time for other threads. Also, multiple 
threads' I/O might bunch up for elevator seeking as Patricia 
describes.

However, threads hit a limit of usefulness if you get too many.
Fortunately that limit should be higher than that imposed by 
your I/O subsystem. You control this by setting your thread 
pool's maximum count and other control factors appropriately.

For the kind of situation you describe I'd predict the 
optimum maximum to be somewhere around two to four times your 
core count.

I don't know whether NIO or thread-based expansion will help 
you more. Even the predictions I am making are implicity 
heavily burdened with caveats to measure and take nothing for 
granted.

I'd likely use a one thread-per-file approach for structural 
reasons, without much initial concern for whether it's the 
best performing. It's a clean architecture that allows good 
locality of working variables and a natural path to a service 
approach. I wouldn't expect any optimization beyond large 
buffers to help individual file performance, so I'd plan for an 
architecture that scales well with increased hardware. One file 
thread per file scales well to one service call per file, 
readily distributed across a cluster. It also fits well with 
expansions to an I/O subsystem.

Amdahl's Law is your friend if you use it to plan units of 
work that have little to no common paths.

-- 
Lew

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web