Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #25748 > unrolled thread

replace extended characters

Back to article view | Back to comp.lang.java.programmer

Contents

  replace extended characters VIDEO MAN <bigmush7@googlemail.com> - 2011-02-10 15:33 -0800
    Re: replace extended characters Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-11 18:40 -0500
    Re: replace extended characters Arne Vajhøj <arne@vajhoej.dk> - 2011-02-10 21:52 -0500
    Re: replace extended characters Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-10 19:37 -0500
    Re: replace extended characters Lew <noone@lewscanon.com> - 2011-02-10 19:18 -0500
      Re: replace extended characters Roedy Green <see_website@mindprod.com.invalid> - 2011-02-11 16:55 -0800
    Re: replace extended characters Owen Jacobson <angrybaldguy@gmail.com> - 2011-02-11 22:15 -0500
    Re: replace extended characters Roedy Green <see_website@mindprod.com.invalid> - 2011-02-11 15:07 -0800
      Re: replace extended characters Roedy Green <see_website@mindprod.com.invalid> - 2011-02-11 15:11 -0800

#25748 — replace extended characters

Hi,

I'm trying to create a java utility that will read in a file that may
or may not contain extended ascii characters and replace these
characters with a predetermined character e.g. replace é with e and
then write the amended file out.

How would people suggest I approach this from an efficiency  point of
view given that the input files could be pretty large?

Any guidance appreciated.

[toc] | [next] | [standalone]

#25896

On 02/11/2011 06:11 PM, Roedy Green wrote:
> My version reads the entire file into RAM in one I/O.  You could
> modify it to read one line of a file at it a time. For the code to do
> that talk to http://mindprod.com/applet/fileio.html
>
> By making whacking huge buffers, you can ensure the bottleneck is the
> CPU.  I use a big switch statement.  For extra speed you could use an
> array lookup of the replacement string for each char.  The
> compiler/JVM  is not all that clever about generating code for switch
> statements.

If the file is very large (on the order of 1GB or larger), you may have 
paging bottlenecks and the mere bottleneck of having to wait for every 
block to be loaded into memory before doing any work, so you spend a 
large amount of time in I/O wait unable to do anything at all. I suspect 
that using mmap'd-like API (i.e., java.nio.MappedByteBuffer) would be 
more efficient, especially if the kernel decides to be nice and preload 
pages without being in page fault.

The Java compiler will make switch statements into a straight jump table 
if it's dense enough, but jump tables can wreak havoc on caches and 
branch predicting, so the JIT may unroll jump tables into better 
constructs at runtime.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#25946

On 10-02-2011 18:33, VIDEO MAN wrote:
> I'm trying to create a java utility that will read in a file that may
> or may not contain extended ascii characters and replace these
> characters with a predetermined character e.g. replace é with e and
> then write the amended file out.
>
> How would people suggest I approach this from an efficiency  point of
> view given that the input files could be pretty large?
>
> Any guidance appreciated.

Most likely there are not so many choices.

You need to decide on IO classes. You should pick
either a class with buffer and not buffer yourself or
a class without buffer and buffer yourself.

If it is single byte charset then you can process
either as bytes or chars otherwise you will need
to process chars.

You can replace either with if/switch or array lookup.

I guess that turned out to be 2 x 2 x 2 = 8
possible designs and in reality they are pretty
similar.

Arne

[toc] | [prev] | [next] | [standalone]

#25961

On 02/10/2011 06:33 PM, VIDEO MAN wrote:
> I'm trying to create a java utility that will read in a file that may
> or may not contain extended ascii characters and replace these
> characters with a predetermined character e.g. replace é with e and
> then write the amended file out.

What charset is the input/output file in? If (AND ONLY IF) you can 
guarantee that it is a fixed-width charset like ISO 8859-1, then you can 
use binary input and use a lookup table to do transformations, probably 
on blocks of 1-4K. If you can't guarantee that, then 
character-by-character reading is the best way to go.

> How would people suggest I approach this from an efficiency  point of
> view given that the input files could be pretty large?

How large is "pretty large"? If you're only talking about a few MB, 
probably any method you come up with is going to be fast. If you're 
talking on the order of hundreds of MB, you'll probably need to go with 
a lookup table approach.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#26025

VIDEO MAN wrote:
> I'm trying to create a java [sic] utility that will read in a file that may
> or may not contain extended ascii [sic] characters and replace these
> characters with a predetermined character [sic] e.g. [sic] replace é with e and
> then write the amended file out.
>
> How would people suggest I approach this from an efficiency  point of
> view given that the input files could be pretty large?
>
> Any guidance appreciated.

Read from a BufferedReader.  Write to a BufferedWriter.  Process one character 
at a time.  It won't be efficient unless you are guaranteed a limited 
character-set input.  The Unicode character space is on the order of 2^24 
characters large.  "Extended ASCII" is a very tiny subset of that, and also 
depends on the character encoding.

If you are certain that the set of possible input characters is small, and 
those you wish to substitute even smaller, you can use a lookup table.  Use a 
'Map<Character,Character>' (will choke on supplementary code points) for 
those, and only those, you wish to substitute.  If the key is absent, pass the 
source character through unchanged.  If present, replace with the associated 
value.

-- 
Lew
Ceci n'est pas une fenêtre.
.___________.
|###] | [###|
|##/  | *\##|
|#/ * |   \#|
|#----|----#|
||    |  * ||
|o *  |    o|
|_____|_____|
|===========|

[toc] | [prev] | [next] | [standalone]

#26193

On Thu, 10 Feb 2011 19:18:36 -0500, Lew <noone@lewscanon.com> wrote,
quoted or indirectly quoted someone who said :

>If you are certain that the set of possible input characters is small, and 
>those you wish to substitute even smaller, you can use a lookup table.  Use a 
>'Map<Character,Character>' (will choke on supplementary code points) for 
>those, and only those, you wish to substitute.  If the key is absent, pass the 
>source character through unchanged.  If present, replace with the associated 
>value.

If you are just handling 256 or 4906 possible chars, then an ordinary
array will be both easy to code and very fast. The highest named
entity is 9830 &diams; 0x2666 black diamond suit.
There are 2^16 = 65536 possible 16-bit unicode chars. Which chars do
you transform?
-- 
Roedy Green Canadian Mind Products
http://mindprod.com
Refactor early. If you procrastinate, you will have
even more code to adjust based on the faulty design.
.

[toc] | [prev] | [next] | [standalone]

#26111

On 2011-02-10 18:33:39 -0500, VIDEO MAN said:

> Hi,
> 
> I'm trying to create a java utility that will read in a file that may
> or may not contain extended ascii characters and replace these
> characters with a predetermined character e.g. replace é with e and
> then write the amended file out.
> 
> How would people suggest I approach this from an efficiency  point of
> view given that the input files could be pretty large?
> 
> Any guidance appreciated.

This process already has a name: "normalization". The Java standard 
library includes tools (java.text.Normalizer and friends) for applying 
the standard Unicode normalizations. A rough sketch for the program 
would be:

1. Load your input text under the correct encoding. You will likely 
want to leave the choice of encodings up to the user; "extended ASCII" 
is not an encoding but encompasses several possible options, and since 
there are only statistical and not deterministically correct ways to 
detect encodings, it's better not to guess.

2. Normalize the text under NFD. This will replace characters like 'ü' 
with a sequence containing a simple character - 'u' in this case - and 
a sequence of combining marks - '¨' in this case. (Alternately, use 
NFKD instead of NFD. NFKD is more liberal about changing the meaning of 
the normalized text, but permits things like detatching ligatures which 
NFD does not do. Examples are in the normalization spec - see figure 6.)

3. Output the resulting normalized text under the target encoding 
(presumably US-ASCII). You'll want to do this the "hard way", via 
java.nio.charset.Charset and CharsetEncoder, so that you can use the 
onUnmappableCharacter action CodingErrorAction.IGNORE to strip 
unencodable characters.

You'll want to read up on normalization and the Unicode normalization 
specs[0] before proceeding. This topic is fraught with non-obvious edge 
cases.

The suggestion that you use iconv or other existing Unicode-aware 
encoding conversion tools is a good one, incidentally. This isn't 
really a problem you need to solve yourself, unless you're completely 
convinced that none of the usual normalization rules is right for your 
use case.

-o

[0] <http://unicode.org/reports/tr15/>

[toc] | [prev] | [next] | [standalone]

#26127

On Thu, 10 Feb 2011 15:33:39 -0800 (PST), VIDEO MAN
<bigmush7@googlemail.com> wrote, quoted or indirectly quoted someone
who said :

>I'm trying to create a java utility that will read in a file that may
>or may not contain extended ascii characters and replace these
>characters with a predetermined character e.g. replace =E9 with e and
>then write the amended file out.
>
>How would people suggest I approach this from an efficiency  point of
>view given that the input files could be pretty large?

Have at look at http://mindprod.com/products1.html#ENTITIES

It includes a program called Entify that finds awkward chars and
replaces them with &xxxx; entities in a set of files. There is also a
program that does the reverse, DeEntify.

You could use the code almost as is and simply modify the table of
entities with your unaccented versions of the chars.


-- 
Roedy Green Canadian Mind Products
http://mindprod.com
Refactor early. If you procrastinate, you will have
even more code to adjust based on the faulty design.
.

[toc] | [prev] | [next] | [standalone]

#26205

On Fri, 11 Feb 2011 15:07:22 -0800, Roedy Green
<see_website@mindprod.com.invalid> wrote, quoted or indirectly quoted
someone who said :

>
>Have at look at http://mindprod.com/products1.html#ENTITIES
>
>It includes a program called Entify that finds awkward chars and
>replaces them with &xxxx; entities in a set of files. There is also a
>program that does the reverse, DeEntify.
>
>You could use the code almost as is and simply modify the table of
>entities with your unaccented versions of the chars.

My version reads the entire file into RAM in one I/O.  You could
modify it to read one line of a file at it a time. For the code to do
that talk to http://mindprod.com/applet/fileio.html 

By making whacking huge buffers, you can ensure the bottleneck is the
CPU.  I use a big switch statement.  For extra speed you could use an
array lookup of the replacement string for each char.  The
compiler/JVM  is not all that clever about generating code for switch
statements.
-- 
Roedy Green Canadian Mind Products
http://mindprod.com
Refactor early. If you procrastinate, you will have
even more code to adjust based on the faulty design.
.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web