Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #28409 > unrolled thread

byte stream vs char stream buffer

Back to article view | Back to comp.lang.java.programmer

Contents

  byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-06 23:48 -0700
    Re: byte stream vs char stream buffer Andreas Leitgeb <avl@auth.logic.tuwien.ac.at> - 2014-05-08 13:30 +0000
      Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-09 09:34 -0700
        Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-10 13:18 -0700
          Re: byte stream vs char stream buffer Robert Klemme <shortcutter@googlemail.com> - 2014-05-10 23:24 +0200
            Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-10 16:08 -0700
              Re: byte stream vs char stream buffer Robert Klemme <shortcutter@googlemail.com> - 2014-05-11 16:02 +0200
                Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-11 11:50 -0700
                  Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-11 12:10 -0700
                  Re: byte stream vs char stream buffer Robert Klemme <shortcutter@googlemail.com> - 2014-05-11 21:34 +0200
                    Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-11 12:55 -0700
                  Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 16:55 -0700
                    Re: byte stream vs char stream buffer markspace <markspace@nospam.nospam> - 2014-05-16 17:36 -0700
                      Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 19:52 -0700
                Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 16:37 -0700
            Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-10 17:21 -0700
              Re: byte stream vs char stream buffer Martin Gregorie <martin@address-in-sig.invalid> - 2014-05-11 09:05 +0000
                Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-12 02:07 -0700
                  Re: byte stream vs char stream buffer Silvio <silvio@internet.com> - 2014-05-12 12:20 +0200
                    Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-16 20:19 -0700
                    Re: byte stream vs char stream buffer Roedy Green <see_website@mindprod.com.invalid> - 2014-05-18 05:28 -0700
                      Re: byte stream vs char stream buffer Silvio <silvio@internet.com> - 2014-05-18 23:54 +0200

Page 1 of 2  [1] 2  Next page →

#28409 — byte stream vs char stream buffer

When you do your lego thing to create a BufferedReader or
BufferedWriter, you get to specify two buffer sizes, one in bytes for
the byte stream and one in chars for the character stream.

Given that you have N bytes to spend on buffers for this file, how
should you allocate them?

I have been told the best ratio is 32Kbytes for every 24K chars.  I
can's see why that would be. I would have thought most should go to
the byte stream for physical i/o. Has anyone ever experimented?
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [next] | [standalone]

#28411

Roedy Green <see_website@mindprod.com.invalid> wrote:
> When you do your lego thing to create a BufferedReader or
> BufferedWriter, you get to specify two buffer sizes, one in bytes for
> the byte stream and one in chars for the character stream.

I'm wondering, if your lego stack is higher than necessary.  Are you
placing BufferedWriter over BufferedOutputStream? I think you can
put the BufferedOutputStream back into the lego bag in that case. ;)

[toc] | [prev] | [next] | [standalone]

#28418

On Thu, 8 May 2014 13:30:15 +0000 (UTC), Andreas Leitgeb
<avl@auth.logic.tuwien.ac.at> wrote, quoted or indirectly quoted
someone who said :

>I'm wondering, if your lego stack is higher than necessary.  Are you
>placing BufferedWriter over BufferedOutputStream? I think you can
>put the BufferedOutputStream back into the lego bag in that case. ;)

I think some experiments are in order.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28424

On Fri, 09 May 2014 09:34:09 -0700, Roedy Green
<see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
someone who said :

>On Thu, 8 May 2014 13:30:15 +0000 (UTC), Andreas Leitgeb
><avl@auth.logic.tuwien.ac.at> wrote, quoted or indirectly quoted
>someone who said :
>
>>I'm wondering, if your lego stack is higher than necessary.  Are you
>>placing BufferedWriter over BufferedOutputStream? I think you can
>>put the BufferedOutputStream back into the lego bag in that case. ;)
>
>I think some experiments are in order.

There are the results:
Using a file of random characters of 400 MB and buffers of 32K bytes

HunkIO 1.20 seconds <-- does an all at once read.
using BufferedReader backed with BufferedInputStream ratio 0.50
buffsize 32768 bytes  1.65 seconds
plain BufferedReader buffsize 32768 bytes 1.76 seconds

The bottom line is either use HunkIO when you have the RAM or give 50%
of your buffer space to the inputstream and 50% to the Reader 
in other works 100K of buffer space would have a 50K inputstream
buffer and the reader would have a 25K buffer because it is measured
in chars.  I tried ratios from .1 to .9.  50% was best.

There is a second problem with plain buffered readers:

final BufferedReader ibr = new BufferedReader( new FileReader(
sampleDataFile ), buffsize / 2 );

There is no place to put the encoding.


The "lego" (encapsulated)  is what you should use for big files:

double ratio = .5;

final FileInputStream fis = new FileInputStream( sampleDataFile );
        final BufferedInputStream bis = new BufferedInputStream( fis,
( int ) ( buffsize * ( 1 - ratio ) ) ); /* buffsize in bytes */
        final InputStreamReader isr = new InputStreamReader( bis,
UTF8Charset );
        final BufferedReader ibr = new BufferedReader( isr, ( int ) (
buffsize * ratio / 2 /* in chars */ ) );

I have an SSD.  I will retest this on a hard drive.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28426

On 10.05.2014 22:18, Roedy Green wrote:
> On Fri, 09 May 2014 09:34:09 -0700, Roedy Green
> <see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
> someone who said :
>
>> On Thu, 8 May 2014 13:30:15 +0000 (UTC), Andreas Leitgeb
>> <avl@auth.logic.tuwien.ac.at> wrote, quoted or indirectly quoted
>> someone who said :
>>
>>> I'm wondering, if your lego stack is higher than necessary.  Are you
>>> placing BufferedWriter over BufferedOutputStream? I think you can
>>> put the BufferedOutputStream back into the lego bag in that case. ;)
>>
>> I think some experiments are in order.
>
> There are the results:
> Using a file of random characters of 400 MB and buffers of 32K bytes
>
> HunkIO 1.20 seconds <-- does an all at once read.
> using BufferedReader backed with BufferedInputStream ratio 0.50
> buffsize 32768 bytes  1.65 seconds
> plain BufferedReader buffsize 32768 bytes 1.76 seconds

I find this a fairly unspecific description of the test you did.  We do 
not know what your test did to ensure file system caching or GC effects 
did not influence the measurement etc.

> The bottom line is either use HunkIO when you have the RAM or give 50%
> of your buffer space to the inputstream and 50% to the Reader
> in other works 100K of buffer space would have a 50K inputstream
> buffer and the reader would have a 25K buffer because it is measured
> in chars.  I tried ratios from .1 to .9.  50% was best.

If I'm not mistaken you did not test cases where there is just a 
BufferedReader OR a BufferedInputStream - that was what Andreas was 
hinting at if I'm not mistaken.  I don't think duplicate buffering will 
make things better.  I had expected at least three tests (double 
buffering, byte buffering, char buffering) plus the code.

Cheers

	robert

[toc] | [prev] | [next] | [standalone]

#28427

On Sat, 10 May 2014 23:24:57 +0200, Robert Klemme
<shortcutter@googlemail.com> wrote, quoted or indirectly quoted
someone who said :

>I find this a fairly unspecific description of the test you did.  We do 
>not know what your test did to ensure file system caching or GC effects 
>did not influence the measurement etc.

the code is posted at
http://mindprod.com/applet/fileio.html#BUFFERSIZE

These are not particularly subtle effects.  I ran five trials and took
the best time for each. I figured this would eliminate hotspot
initialisation and GC effects.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28435

On 11.05.2014 01:08, Roedy Green wrote:
> On Sat, 10 May 2014 23:24:57 +0200, Robert Klemme
> <shortcutter@googlemail.com> wrote, quoted or indirectly quoted
> someone who said :
>
>> I find this a fairly unspecific description of the test you did.  We do
>> not know what your test did to ensure file system caching or GC effects
>> did not influence the measurement etc.
>
> the code is posted at
> http://mindprod.com/applet/fileio.html#BUFFERSIZE

I followed a slightly different approach and modularized my test a bit 
more.  You can find it here:

https://gist.github.com/rklemme/fad399b5e7cc4d3b6d0c

File opening and closing is not measured.  You can assume the data to 
purely come from memory (vastly more mem than file size + no disk 
activity shown).  This was on an Xbuntu 14.04 64 bit, kernel version 
3.13.0-24.  No special JVM command line switches were used.

> These are not particularly subtle effects.  I ran five trials and took
> the best time for each. I figured this would eliminate hotspot
> initialisation and GC effects.

Sounds good.  I followed the same approach (i.e. repeating and using the 
min time value).

My results:

  - The higher level the buffering, the more effective.
  - Buffering on byte level often actually rather hurts than helps.
  - Reading in 1k chunks from the application level is already close to 
optimal.

Kind regards

	robert

[toc] | [prev] | [next] | [standalone]

#28436

On 5/11/2014 7:02 AM, Robert Klemme wrote:

>   - Reading in 1k chunks from the application level is already close to
> optimal.


Good test suit here all around, Robert.  I just wanted address this one 
point quickly.  I'm not sure if this is new information or not, but in 
actual use, I've found that it helps to allocate large buffers when 
you're reading a large text file that must be processed quickly.

Sample:

       ByteBuffer bb = ByteBuffer.allocateDirect( 64 * 1024 );
       RandomAccessFile f = new RandomAccessFile( "blet", "rw" );
       FileChannel fc = f.getChannel();
       readToCapacity( fc, bb );
       bb.flip();
       ByteBuffer b2 = bb.slice();
       b2.asReadOnlyBuffer();

Allocating a direct byte buffer of 64k seems to help text processing 
speed by about an order of magnitude.  Files that take minutes to 
process go to seconds when large direct byte buffers are used.  It seems 
important to get that first big chunk of bytes in before doing anything 
else (like converting to chars).

The input file for this was about 600k, so if you're reading much 
smaller files the difference might not show up.

This is just a sample, I don't seem to be able to find the original, so 
I hope it's accurate.  Here's the static method used by the above code:

    public static void readToCapacity( FileChannel fc, ByteBuffer buf )
            throws IOException
    {
       final int capacity = buf.capacity();
       int totalRead = 0;
       int bytesRead = 0;
       while( (bytesRead = fc.read( buf ) ) != -1 ) {
          totalRead += bytesRead;
          if( totalRead == capacity ) break;
       }
    }

[toc] | [prev] | [next] | [standalone]

#28437

I think this might be the original code that I used to speed up my 
project.  It's been a while so I don't know exactly how well this is 
working right now.

You can wrap this in whatever you'd like -- BufferedRead for example -- 
since I just make a plain InputStream here.  I also notice that there's 
not even a close method on this class, so obviously I didn't complete it.

import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;

/**
  * Copyright 2013 Brenden Towey DONATED TO THE PUBLIC DOMAIN.
  *
  * PROVIDED AS-IS WITH NO WARRANTY.  FURTHER TESTING IS
  * REQUIRED TO COMPLETE THIS CODE.  CAVEAT EMPTOR.
  *
  * @author Brenden Towey
  */
public class FileByteBufferInputStream extends InputStream {

    private ByteBuffer buffer = ByteBuffer.allocateDirect( 64 * 1024 );
    private FileInputStream file;
    int read;
    boolean eof = false;

    public FileByteBufferInputStream( FileInputStream file ) throws 
IOException
    {
       this.file = file;
       read = file.getChannel().read( buffer );
       if( read == -1 ) eof = true;
       buffer.rewind();
    }

    @Override
    public int read()
            throws IOException
    {
       if( eof ) return -1;
       if( read-- == 0 )
       {
          buffer.clear();
          read = file.getChannel().read( buffer );
          if( read == -1 ) eof = true;
          buffer.rewind();
          if( eof ) {
             return -1;
          } else {
             int retval = buffer.get();
             assert( read > 0 );
             read--;
             return retval;
          }
       } else{
          int retval = buffer.get();
          return retval;
       }
    }
}

[toc] | [prev] | [next] | [standalone]

#28438

On 11.05.2014 20:50, markspace wrote:
> On 5/11/2014 7:02 AM, Robert Klemme wrote:
>
>>   - Reading in 1k chunks from the application level is already close to
>> optimal.
>
>
> Good test suit here all around, Robert.

Thank you!

>  I just wanted address this one
> point quickly.  I'm not sure if this is new information or not, but in
> actual use, I've found that it helps to allocate large buffers when
> you're reading a large text file that must be processed quickly.
>
> Sample:
>
>        ByteBuffer bb = ByteBuffer.allocateDirect( 64 * 1024 );
>        RandomAccessFile f = new RandomAccessFile( "blet", "rw" );
>        FileChannel fc = f.getChannel();
>        readToCapacity( fc, bb );
>        bb.flip();
>        ByteBuffer b2 = bb.slice();
>        b2.asReadOnlyBuffer();
>
> Allocating a direct byte buffer of 64k seems to help text processing
> speed by about an order of magnitude.  Files that take minutes to
> process go to seconds when large direct byte buffers are used.  It seems
> important to get that first big chunk of bytes in before doing anything
> else (like converting to chars).

I seem to recall having done similar tests in the past but found that 
overall the benefit for processing speed was negligible because at some 
point the data has to be moved into the Java heap.  But my memory is 
faint here and the situation may have changed in the meantime.

> The input file for this was about 600k, so if you're reading much
> smaller files the difference might not show up.
>
> This is just a sample, I don't seem to be able to find the original, so
> I hope it's accurate.  Here's the static method used by the above code:
>
>     public static void readToCapacity( FileChannel fc, ByteBuffer buf )
>             throws IOException
>     {
>        final int capacity = buf.capacity();
>        int totalRead = 0;
>        int bytesRead = 0;
>        while( (bytesRead = fc.read( buf ) ) != -1 ) {
>           totalRead += bytesRead;
>           if( totalRead == capacity ) break;
>        }
>     }

The code shown so far has some differences to the tests Roedy and I were 
doing:

1. You are never transferring those bytes to Java land (i.e. into a byte 
or byte[] which is allocated on the heap) - data stays in native land.

2. You are not reading chars and hence also do not do the character 
decoding.

You likely did some processing but we cannot see it.  But I agree, 
another test with nio would be in order to show whether there is more 
potential for speedup.

Btw. a memory mapped file may also be an alternative.  This can be done 
with nio.  It does have some downsides (e.g. different behavior on 
different OS in my experience, address space) and proved not to be 
faster than plain sequential reading with InputStreams in an application 
I have been working on.

Kind regards

	robert

[toc] | [prev] | [next] | [standalone]

#28439

On 5/11/2014 12:34 PM, Robert Klemme wrote:

> 1. You are never transferring those bytes to Java land (i.e. into a byte
> or byte[] which is allocated on the heap) - data stays in native land.
>
> 2. You are not reading chars and hence also do not do the character
> decoding.


Yes, I knew the data was "mostly ascii" and therefore I didn't have to 
do character decoding.  An efficient UTF-8 converter shouldn't be much 
more complicated however.

I appear to be counting word lengths in the file, I'm not sure why at 
this point.  Some more found code:

          FileInputStream fins = new FileInputStream( path.toFile() );
          FileByteBufferInputStream fbbins =
                  new FileByteBufferInputStream( fins );

          int charRead;
          HashedHistogram histogram = new HashedHistogram();
          charRead = fbbins.read();
          StringBuilder sb = new StringBuilder();
          while( charRead != -1 )
          {
             if( charRead < 128 && !Character.isWhitespace( charRead ) ) {
                sb.append( (char) charRead );
                charRead = fbbins.read();
             } else {
                histogram.add( sb.toString() );
                sb.delete( 0, sb.length() );
                while( (Character.isWhitespace( (charRead = 
fbbins.read() )) ||
                        charRead >= 128) && charRead != -1 )
                {
                   // nothing
                }
             }
          }
          System.out.println( histogram.size() + " words" );
          Entry<Comparable,Integer>[] entries = 
histogram.getSortedEntries();
          System.out.println( "Bottom words:" );
          for( int i = 0; i < 20; i++ )
             System.out.println( entries[i].getKey()+", 
"+entries[i].getValue() );
          System.out.println( "Top words:" );
          for( int i = entries.length-1; i > entries.length-41; i-- )
             System.out.println( entries[i].getKey()+", 
"+entries[i].getValue() );


Kind of ugly, but that's what I have.

[toc] | [prev] | [next] | [standalone]

#28500

On Sun, 11 May 2014 11:50:24 -0700, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>       ByteBuffer bb = ByteBuffer.allocateDirect( 64 * 1024 );
>       RandomAccessFile f = new RandomAccessFile( "blet", "rw" );
>       FileChannel fc = f.getChannel();
>       readToCapacity( fc, bb );
>       bb.flip();
>       ByteBuffer b2 = bb.slice();
>       b2.asReadOnlyBuffer();

I would like this as an example of nio. I wonder why each step is
necessary.


// sample use of NIO to read a text file very rapidly
// by Mark Space

// alloc buffer suited for direct physical I/O
ByteBuffer bb = ByteBuffer.allocateDirect( 64 * 1024 ); 

RandomAccessFile f = new RandomAccessFile( "blet", "rw" );

FileChannel fc = f.getChannel();

// A user method to consume one chunk of data
// It would have to convert bytes to chars, fretting over boundaries.
readToCapacity( fc, bb ); 

// request the next chunk of data
bb.flip();

// ? 
ByteBuffer b2 = bb.slice();  

// creates readonly version and discards
b2.asReadOnlyBuffer(); 

-- 
Roedy Green Canadian Mind Products http://mindprod.com
Young man, in mathematics you don’t understand things.
You just get used to them. 
~ John von Neumann (born: 1903-12-28 died: 1957-02-08 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28502

On 5/16/2014 4:55 PM, Roedy Green wrote:

> // ?
> ByteBuffer b2 = bb.slice();

Yes, this bit of code was an amalgam of other bits that I had saved to 
remind myself how to perform some common operations on ByteBuffers.  The 
longer sequence went something like:

       readToCapacity( fc, bb );
       bb.flip();
       ByteBuffer b2 = bb.slice();
       b2.limit( 100 );
       b2.asReadOnlyBuffer();
       readHeader( b2 );
       readFile( bb );

And it was just a way to be able to isolate the (100 byte) header from 
the rest of the stream, as well as be able to retain all original bytes 
without a potentially expensive seek operation.  Mostly just as a 
reminder how to slice up a ByteBuffer into smaller buffers so they can 
be read independently.

 > // request the next chunk of data
 > bb.flip();
 >

BTW, I'd make that comment "reset limit and position for reading" rather 
than "next chuck" (the next chunk would probably be clear(), because if 
the buffer isn't full you'd run into problems with limit using flip()).

Flip() is used to switch from put()s to get()s.  See the docs:

http://docs.oracle.com/javase/7/docs/api/java/nio/Buffer.html#flip()

[toc] | [prev] | [next] | [standalone]

#28506

On Fri, 16 May 2014 17:36:31 -0700, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>lip() is used to switch from put()s to get()s.  See the docs:
>
>http://docs.oracle.com/javase/7/docs/api/java/nio/Buffer.html#flip()

I read that but those operations have nothing to do with the
traditional notions of I/O.

I have tried to make sense of this and figure out what it does that
ordinary I/O does not, especially since it legos on top of ordinary
I/O. I have wimped out twice before.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Young man, in mathematics you don’t understand things.
You just get used to them. 
~ John von Neumann (born: 1903-12-28 died: 1957-02-08 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28497

On Sun, 11 May 2014 16:02:40 +0200, Robert Klemme
<shortcutter@googlemail.com> wrote, quoted or indirectly quoted
someone who said :

>
>  - The higher level the buffering, the more effective.

You are doing your own buffering, not using BufferedReader right?

In that case, when you read a char[] it might do a big physical i/o,
no?

In the days of 16K computers, when you did this sort of thing, the OS
would schedule a read to a second buffer while you were processing the
first. Oddly now that we have RAM to burn, we no longer do that though
I suppose the OS might do some speculative reading.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Young man, in mathematics you don’t understand things.
You just get used to them. 
~ John von Neumann (born: 1903-12-28 died: 1957-02-08 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28428

On Sat, 10 May 2014 23:24:57 +0200, Robert Klemme
<shortcutter@googlemail.com> wrote, quoted or indirectly quoted
someone who said :

>I don't think duplicate buffering will 
>make things better.

It does. And allocating your space exactly 50:50 makes a noticeable
difference.

the plain Buffered reader was done like this:

final BufferedReader ibr = new BufferedReader( new FileReader(
sampleDataFile ), buffsize / 2 );

It got whatever the default buffering on the internal inputstream is.
It is not as fast as specifying buffers for the internal inputstream,
lego-style.


-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28433

On Sat, 10 May 2014 17:21:19 -0700, Roedy Green wrote:

> On Sat, 10 May 2014 23:24:57 +0200, Robert Klemme
> <shortcutter@googlemail.com> wrote, quoted or indirectly quoted someone
> who said :
> 
>>I don't think duplicate buffering will make things better.
> 
> It does. And allocating your space exactly 50:50 makes a noticeable
> difference.
> 
> the plain Buffered reader was done like this:
> 
> final BufferedReader ibr = new BufferedReader( new FileReader(
> sampleDataFile ), buffsize / 2 );
>
You said earlier that you were partitioning a fixed amount of buffer 
space between the BufferedInputStream and the BufferedReader, but that's 
not what your code shows.

Instead the BufferedInputStream  is getting part of a 'buffer space 
allocation', which is set to 32K. This part varies from 10% to 90% of the 
32K while the BufferedReader always gets 16K. This means that the total 
buffer space used by the combination of BufferedInputStream and 
BufferedReader varies from 19.2 KB to 60.8 KB as the 'ratio' varies from 
0.1 to 0.9. 

In other words, you're not measuring what you thought you're measuring 
because the total buffer space used by each test varies by a factor of 
three during the tests and will almost certainly affect your results.



-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#28445

On Sun, 11 May 2014 09:05:16 +0000 (UTC), Martin Gregorie
<martin@address-in-sig.invalid> wrote, quoted or indirectly quoted
someone who said :

>
>Instead the BufferedInputStream  is getting part of a 'buffer space 
>allocation', which is set to 32K. This part varies from 10% to 90% of the 
>32K while the BufferedReader always gets 16K.


    private static void withBufferedReader( double ratio, int buffsize
) throws IOException
        {
        assert .1 <= ratio && ratio <= .9 : "ratio must be in range
[0.1,0.9].";
        long prevTime = System.nanoTime();
        final FileInputStream fis = new FileInputStream(
sampleDataFile );
        final BufferedInputStream bis = new BufferedInputStream( fis,
( int ) ( buffsize * ( 1 - ratio ) ) ); /* buffsize in bytes */
        final InputStreamReader isr = new InputStreamReader( bis,
EIO.UTF8Charset );
        final BufferedReader ibr = new BufferedReader( isr, ( int ) (
buffsize * ratio / 2 /* in chars */ ) );


so for example if ratio = .5 and buffsize =32K,
then the stream will get 16K bytes and the reader will get 8k chars.

This is the optimum ratio.

-- 
Roedy Green Canadian Mind Products http://mindprod.com
Culture is your operating system. 
~ Terence McKenna (born: 1946-11-16 died: 2000-04-03 at age: 53)

[toc] | [prev] | [next] | [standalone]

#28446

On 05/12/2014 11:07 AM, Roedy Green wrote:
> On Sun, 11 May 2014 09:05:16 +0000 (UTC), Martin Gregorie
> <martin@address-in-sig.invalid> wrote, quoted or indirectly quoted
> someone who said :
>
>>
>> Instead the BufferedInputStream  is getting part of a 'buffer space
>> allocation', which is set to 32K. This part varies from 10% to 90% of the
>> 32K while the BufferedReader always gets 16K.
>
>
>      private static void withBufferedReader( double ratio, int buffsize
> ) throws IOException
>          {
>          assert .1 <= ratio && ratio <= .9 : "ratio must be in range
> [0.1,0.9].";
>          long prevTime = System.nanoTime();
>          final FileInputStream fis = new FileInputStream(
> sampleDataFile );
>          final BufferedInputStream bis = new BufferedInputStream( fis,
> ( int ) ( buffsize * ( 1 - ratio ) ) ); /* buffsize in bytes */
>          final InputStreamReader isr = new InputStreamReader( bis,
> EIO.UTF8Charset );
>          final BufferedReader ibr = new BufferedReader( isr, ( int ) (
> buffsize * ratio / 2 /* in chars */ ) );
>
>
> so for example if ratio = .5 and buffsize =32K,
> then the stream will get 16K bytes and the reader will get 8k chars.
>
> This is the optimum ratio.
>

Roedy,

It would be interesting to repeat this experiment with different 
character encodings and see if using UTF-16 versus UTF-8 makes a 
difference here.

Silvio

[toc] | [prev] | [next] | [standalone]

#28507

On Mon, 12 May 2014 12:20:11 +0200, Silvio <silvio@internet.com>
wrote, quoted or indirectly quoted someone who said :

>It would be interesting to repeat this experiment with different 
>character encodings and see if using UTF-16 versus UTF-8 makes a 
>difference here.

With UTF-16 there are twice as many byte to read, but the 50% magic
ratio still prevails.

best 5 five trials shown.

Using a random sample data file of 209,715,200 chars 419,430,402
bytes.
Using aggregate buffersize of 65,536 bytes.
Using charset UTF-16

BufferedReader backed with BufferedInputStream ratio 0.10 buffsize
65536 bytes  2.64 seconds
BufferedReader backed with BufferedInputStream ratio 0.20 buffsize
65536 bytes  2.63 seconds
BufferedReader backed with BufferedInputStream ratio 0.30 buffsize
65536 bytes  2.64 seconds
BufferedReader backed with BufferedInputStream ratio 0.40 buffsize
65536 bytes  2.64 seconds
BufferedReader backed with BufferedInputStream ratio 0.50 buffsize <--
65536 bytes  2.64 seconds
BufferedReader backed with BufferedInputStream ratio 0.60 buffsize
65536 bytes  2.68 seconds
BufferedReader backed with BufferedInputStream ratio 0.70 buffsize
65536 bytes  2.71 seconds
BufferedReader backed with BufferedInputStream ratio 0.80 buffsize
65536 bytes  2.79 seconds
BufferedReader backed with BufferedInputStream ratio 0.90 buffsize
65536 bytes  2.70 seconds
HunkIO 2.70 seconds





Using a random sample data file of 209,715,200 chars 209,715,200
bytes.
Using aggregate buffersize of 65,536 bytes.
Using charset UTF-8
HunkIO 0.73 seconds
BufferedReader backed with BufferedInputStream ratio 0.10 buffsize
65536 bytes  0.89 seconds
BufferedReader backed with BufferedInputStream ratio 0.20 buffsize
65536 bytes  0.88 seconds
BufferedReader backed with BufferedInputStream ratio 0.30 buffsize
65536 bytes  0.88 seconds
BufferedReader backed with BufferedInputStream ratio 0.40 buffsize
65536 bytes  0.89 seconds
BufferedReader backed with BufferedInputStream ratio 0.50 buffsize <--
65536 bytes  0.83 seconds
BufferedReader backed with BufferedInputStream ratio 0.60 buffsize
65536 bytes  0.91 seconds
BufferedReader backed with BufferedInputStream ratio 0.70 buffsize
65536 bytes  0.92 seconds
BufferedReader backed with BufferedInputStream ratio 0.80 buffsize
65536 bytes  0.96 seconds
BufferedReader backed with BufferedInputStream ratio 0.90 buffsize
65536 bytes  0.92 seconds

There is something strange. UTF-16 should be faster to convert to
Strings, and at worst, taking twice as long for physical I/O.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Young man, in mathematics you don’t understand things.
You just get used to them. 
~ John von Neumann (born: 1903-12-28 died: 1957-02-08 at age: 53)

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.java.programmer

csiph-web