Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #28436

Re: byte stream vs char stream buffer

Show all headers | View raw

On 5/11/2014 7:02 AM, Robert Klemme wrote:

>   - Reading in 1k chunks from the application level is already close to
> optimal.


Good test suit here all around, Robert.  I just wanted address this one 
point quickly.  I'm not sure if this is new information or not, but in 
actual use, I've found that it helps to allocate large buffers when 
you're reading a large text file that must be processed quickly.

Sample:

       ByteBuffer bb = ByteBuffer.allocateDirect( 64 * 1024 );
       RandomAccessFile f = new RandomAccessFile( "blet", "rw" );
       FileChannel fc = f.getChannel();
       readToCapacity( fc, bb );
       bb.flip();
       ByteBuffer b2 = bb.slice();
       b2.asReadOnlyBuffer();

Allocating a direct byte buffer of 64k seems to help text processing 
speed by about an order of magnitude.  Files that take minutes to 
process go to seconds when large direct byte buffers are used.  It seems 
important to get that first big chunk of bytes in before doing anything 
else (like converting to chars).

The input file for this was about 600k, so if you're reading much 
smaller files the difference might not show up.

This is just a sample, I don't seem to be able to find the original, so 
I hope it's accurate.  Here's the static method used by the above code:

    public static void readToCapacity( FileChannel fc, ByteBuffer buf )
            throws IOException
    {
       final int capacity = buf.capacity();
       int totalRead = 0;
       int bytesRead = 0;
       while( (bytesRead = fc.read( buf ) ) != -1 ) {
          totalRead += bytesRead;
          if( totalRead == capacity ) break;
       }
    }

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-06 23:48 -0700
  Re: byte stream vs char stream buffer Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-05-08 13:30 +0000
    Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-09 09:34 -0700
      Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-10 13:18 -0700
        Re: byte stream vs char stream buffer Robert Klemme <shortcutter@googlemail.com> - 2014-05-10 23:24 +0200
          Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-10 16:08 -0700
            Re: byte stream vs char stream buffer Robert Klemme <shortcutter@googlemail.com> - 2014-05-11 16:02 +0200
              Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-11 11:50 -0700
                Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-11 12:10 -0700
                Re: byte stream vs char stream buffer Robert Klemme <shortcutter@googlemail.com> - 2014-05-11 21:34 +0200
                Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-11 12:55 -0700
                Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 16:55 -0700
                Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-16 17:36 -0700
                Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 19:52 -0700
              Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 16:37 -0700
          Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-10 17:21 -0700
            Re: byte stream vs char stream buffer Martin Gregorie <martin@address-in-sig.invalid> - 2014-05-11 09:05 +0000
              Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-12 02:07 -0700
                Re: byte stream vs char stream buffer Silvio <silvio@internet.com> - 2014-05-12 12:20 +0200
                Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 20:19 -0700
                Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-18 05:28 -0700
                Re: byte stream vs char stream buffer Silvio <silvio@internet.com> - 2014-05-18 23:54 +0200

csiph-web