Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!rt.uk.eu.org!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: markspace <markspace@nospam.nospam>
Newsgroups: comp.lang.java.programmer
Subject: Re: byte stream vs char stream buffer
Date: Sun, 11 May 2014 11:50:24 -0700
Organization: A noiseless patient Spider
Lines: 46
Message-ID: <lkogpm$5f4$1@dont-email.me>
References: <vhljm9llt6jau94jqr573ccpjcnhkunsil@4ax.com> <slrnlmn1n7.bi9.avl@login.logic.tuwien.ac.at> <mr0qm9hbbeqj0rggqco3umj91bqvjeti58@4ax.com> <en0tm9pu331ts50galimgbk8bkseu83mtk@4ax.com> <bt7jpaFa8d5U1@mid.individual.net> <pqbtm9h82nb1q0qbdf87bae2h1jeot5uno@4ax.com> <bt9e82Fll8eU1@mid.individual.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 11 May 2014 18:50:31 +0000 (UTC)
Injection-Info: mx05.eternal-september.org; posting-host="9358cda03e1e0dfd3c4b757daa608707"; logging-data="5604"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/0tSc3LuMEeKl80yBiH17nBxq8Z/qIUCk="
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
In-Reply-To: <bt9e82Fll8eU1@mid.individual.net>
Cancel-Lock: sha1:heaDLLKWf++ms1Kgx0WYMGab3No=
Xref: csiph.com comp.lang.java.programmer:28436

On 5/11/2014 7:02 AM, Robert Klemme wrote:

>   - Reading in 1k chunks from the application level is already close to
> optimal.


Good test suit here all around, Robert.  I just wanted address this one 
point quickly.  I'm not sure if this is new information or not, but in 
actual use, I've found that it helps to allocate large buffers when 
you're reading a large text file that must be processed quickly.

Sample:

       ByteBuffer bb = ByteBuffer.allocateDirect( 64 * 1024 );
       RandomAccessFile f = new RandomAccessFile( "blet", "rw" );
       FileChannel fc = f.getChannel();
       readToCapacity( fc, bb );
       bb.flip();
       ByteBuffer b2 = bb.slice();
       b2.asReadOnlyBuffer();

Allocating a direct byte buffer of 64k seems to help text processing 
speed by about an order of magnitude.  Files that take minutes to 
process go to seconds when large direct byte buffers are used.  It seems 
important to get that first big chunk of bytes in before doing anything 
else (like converting to chars).

The input file for this was about 600k, so if you're reading much 
smaller files the difference might not show up.

This is just a sample, I don't seem to be able to find the original, so 
I hope it's accurate.  Here's the static method used by the above code:

    public static void readToCapacity( FileChannel fc, ByteBuffer buf )
            throws IOException
    {
       final int capacity = buf.capacity();
       int totalRead = 0;
       int bytesRead = 0;
       while( (bytesRead = fc.read( buf ) ) != -1 ) {
          totalRead += bytesRead;
          if( totalRead == capacity ) break;
       }
    }