Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!us.feeder.erje.net!news2.arglkargh.de!news.swapon.de!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: markspace <markspace@nospam.nospam>
Newsgroups: comp.lang.java.programmer
Subject: Re: byte stream vs char stream buffer
Date: Sun, 11 May 2014 12:55:27 -0700
Organization: A noiseless patient Spider
Lines: 55
Message-ID: <lkokjk$3rn$1@dont-email.me>
References: <vhljm9llt6jau94jqr573ccpjcnhkunsil@4ax.com> <slrnlmn1n7.bi9.avl@login.logic.tuwien.ac.at> <mr0qm9hbbeqj0rggqco3umj91bqvjeti58@4ax.com> <en0tm9pu331ts50galimgbk8bkseu83mtk@4ax.com> <bt7jpaFa8d5U1@mid.individual.net> <pqbtm9h82nb1q0qbdf87bae2h1jeot5uno@4ax.com> <bt9e82Fll8eU1@mid.individual.net> <lkogpm$5f4$1@dont-email.me> <bta1moFpubcU1@mid.individual.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 11 May 2014 19:55:33 +0000 (UTC)
Injection-Info: mx05.eternal-september.org; posting-host="9358cda03e1e0dfd3c4b757daa608707"; logging-data="3959"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX18iczaqREWgs65ebOlk/W8EQxH81NhaEcc="
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
In-Reply-To: <bta1moFpubcU1@mid.individual.net>
Cancel-Lock: sha1:hMsHnBkwY0wya/y/u7kCR7llTgA=
Xref: csiph.com comp.lang.java.programmer:28439

On 5/11/2014 12:34 PM, Robert Klemme wrote:

> 1. You are never transferring those bytes to Java land (i.e. into a byte
> or byte[] which is allocated on the heap) - data stays in native land.
>
> 2. You are not reading chars and hence also do not do the character
> decoding.


Yes, I knew the data was "mostly ascii" and therefore I didn't have to 
do character decoding.  An efficient UTF-8 converter shouldn't be much 
more complicated however.

I appear to be counting word lengths in the file, I'm not sure why at 
this point.  Some more found code:

          FileInputStream fins = new FileInputStream( path.toFile() );
          FileByteBufferInputStream fbbins =
                  new FileByteBufferInputStream( fins );

          int charRead;
          HashedHistogram histogram = new HashedHistogram();
          charRead = fbbins.read();
          StringBuilder sb = new StringBuilder();
          while( charRead != -1 )
          {
             if( charRead < 128 && !Character.isWhitespace( charRead ) ) {
                sb.append( (char) charRead );
                charRead = fbbins.read();
             } else {
                histogram.add( sb.toString() );
                sb.delete( 0, sb.length() );
                while( (Character.isWhitespace( (charRead = 
fbbins.read() )) ||
                        charRead >= 128) && charRead != -1 )
                {
                   // nothing
                }
             }
          }
          System.out.println( histogram.size() + " words" );
          Entry<Comparable,Integer>[] entries = 
histogram.getSortedEntries();
          System.out.println( "Bottom words:" );
          for( int i = 0; i < 20; i++ )
             System.out.println( entries[i].getKey()+", 
"+entries[i].getValue() );
          System.out.println( "Top words:" );
          for( int i = entries.length-1; i > entries.length-41; i-- )
             System.out.println( entries[i].getKey()+", 
"+entries[i].getValue() );


Kind of ugly, but that's what I have.