Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #224912
| From | davidson <davidson@freevolt.org> |
|---|---|
| Newsgroups | linux.debian.user |
| Subject | how u mine 4 utf8 [was Re: Using .XCompose] |
| Date | 2020-07-18 03:20 +0200 |
| Message-ID | <AtJG9-3fx-5@gated-at.bofh.it> (permalink) |
| References | (8 earlier) <Arjd7-4cJ-9@gated-at.bofh.it> <ArkVz-5gw-1@gated-at.bofh.it> <ArLCp-4oG-3@gated-at.bofh.it> <AsZ0B-7St-5@gated-at.bofh.it> <AtgXw-1P5-13@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
[Multipart message — attachments visible in raw view] - view raw
On Thu, 16 Jul 2020 Ajith R wrote:
> On Thursday 16 July 2020 4:54:09 AM IST davidson wrote:
[snip]
>> $ sed 'y/\xc2\xa0/%/' somefile
>
> An off topic question: of sed, awk and perl, if I am to chose one to
> learn, which would you suggest. I wanted to do some substitutions. I
> read about them and decided on PERL because from what I understood,
> it has better support for regular expressions and do almost
> everything that sed and awk could do. Have I made the right
> decision?
[interesting question. postponing my answer]
>> (However, one advantage of using the C-style byte-constants (\xHH)
>> instead is that it is easy for everyone to see what they are, the
>> web archive won't replace them with normal spaces, etc.)
>
> Using the Unicode sequence also gives the same advantages, doesn't
> it?
Sure. The unicode code points for characters are a slightly more
abstract value for "what they are" than whatever bytes represent them
in some file, it seems to me. Depending on the purpose, this could be
a good difference.
> I find it difficult to get the "translation" between the unicode
> code values and the hexadecimal/octal representation.
In this case "the hexadecimal/octal representation" is the UTF-8
representation of unicode code points.
My understanding of text handling has a lot of weaknesses and fuzzy
spots that these discussions have made more apparent to me. And the
map from unicode code points to UTF-8 is one of them.
Or at least it was, until yesterday.
I recommend two references I found helpful, though I didn't read
either one especially carefully. I mostly just stared at the templates
(the UTF-8 fighters below), repeatedly asked myself "why did they do
it that way?" and then skimmed to confirm guesses.
First, utf-8(7).
$ man 7 utf-8
Second, Wikipedia has a not-bad article about UTF-8.
Anyways, here is my ELI5/explainer how to represent a unicode code
point in UTF-8.
Mapping unicode to UTF-8
[If you don't/can't/won't display this with a fixed-width font, like
courier or something, you might prefer to stop reading here. The
references above are pretty good anyways.]
It helps me to notice that the UTF-8 encoding of unicode permits you
to know --whenever you grab some octet of bits-- where that byte
belongs in the following triage based on its Most Significant Bits
(MSBs).
The octet...
1. represents an entire character on its own (MSB is 0)
2. is a non-initial byte of a multi-byte character (MSBs are 10)
3. is the initial byte of an N-byte character with N>1 (MSBs are N 1s trailed by a 0)
(Notice that you could rephrase (1-3) above in terms of the position
of the most significant zero bit.)
So given a unicode code point, do this:
1. discard its leading 0 bits,
2. count how many bits you have left (which is how many x you will
need from the templates below), and
3. pick your UTF-8 fighter:
1 octet (7 bits) 0 x x x x x x x
2 octets (11 bits) 1 1 0 x x x x x 1 0 x x x x x x
3 octets (16 bits) 1 1 1 0 x x x x 1 0 x x x x x x 1 0 x x x x x x
4 octets (21 bits) 1 1 1 1 0 x x x 1 0 x x x x x x 1 0 x x x x x x 1 0 x x x x x x
Plug the unicode code point's bits into the rightward slots marked
with x in the template, and fill in any remaining more significant
(leftward) bits with zeros.
Synopsis of worked examples below:
[unicode code point]
[unicode representation, stripped of leading zero bits]
[smallest appropriate UTF-8 template]
[UTF-8 representation]
Example 1. Nonbreaking space character
U+00A0
A 0
1 0 1 0 0 0 0 0
1 1 0 x x x x x 1 0 x x x x x x
C 2 A 0
1 1 0 0 0 0 1 0 1 0 1 0 0 0 0 0
Example 2. Don't know the name of this one. I tell myself it
represents a velar nasal stop-initial syllable, but I would appreciate
correction.
U+0D19
1 1 0 1 0 0 0 1 1 0 0 1
So, we need to fit the 12 bits above into one of the four UTF-8
templates.
The smallest two can only fit 7 and 11 bits respectively, so we'll
need the next larger template, the three-octet one:
1 1 1 0 x x x x 1 0 x x x x x x 1 0 x x x x x x
E 0 B 4 9 9
1 1 1 0 x x x x 1 0 1 1 0 1 0 0 1 0 0 1 1 0 0 1
Example 3. a malayalam virama (sp?)
U+0D4D
1 1 0 1 0 1 0 0 1 1 0 1
1 1 1 0 x x x x 1 0 x x x x x x 1 0 x x x x x x
E 0 B 5 8 D
1 1 1 0 0 0 0 0 1 0 1 1 0 1 0 1 1 0 0 0 1 1 0 1
--
Ce qui est important est rarement urgent
et ce qui est urgent est rarement important
-- Dwight David Eisenhower
Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-06 07:40 +0200
Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-06 14:30 +0200
Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-06 14:50 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-06 21:20 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-06 22:40 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-07 17:20 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-07 08:40 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-07 17:20 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-09 23:50 +0200
Re: Using .XCompose [resend of xev_output_annotated.txt] davidson <davidson@freevolt.org> - 2020-07-10 03:40 +0200
Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-10 10:50 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 10:40 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 12:30 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-12 17:00 +0200
Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-13 06:00 +0200
Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-13 14:00 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-13 20:30 +0200
Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-13 20:40 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-13 20:50 +0200
Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-13 21:50 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-14 05:40 +0200
Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-15 16:50 +0200
Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-15 17:10 +0200
Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-15 17:20 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 01:30 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 02:40 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-16 20:40 +0200
how u mine 4 utf8 [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-18 03:20 +0200
UTF-8 Everywhere -- Re: how u mine 4 utf8 [was Re: Using .XCompose] Zenaan Harkness <zenaan@freedbms.net> - 2020-07-18 07:40 +0200
Re: how u mine 4 utf8 [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-18 09:20 +0200
Re: how u mine 4 utf8 [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 13:00 +0200
Fw: how u mine 4 utf8 [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 12:50 +0200
Re: Fw: how u mine 4 utf8 [was Re: Using .XCompose] Tom Browder <tom.browder@gmail.com> - 2020-07-19 13:10 +0200
Re: how u mine 4 utf8 [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 13:50 +0200
sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-18 12:10 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Zenaan Harkness <zenaan@freedbms.net> - 2020-07-18 12:50 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] William Michels <wjm1@caa.columbia.edu> - 2020-07-18 22:00 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Andrei POPESCU <andreimpopescu@gmail.com> - 2020-07-18 13:20 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] songbird <songbird@anthive.com> - 2020-07-18 21:00 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] <tomas@tuxteam.de> - 2020-07-18 16:00 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] mick crane <mick.crane@gmail.com> - 2020-07-18 22:40 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] <tomas@tuxteam.de> - 2020-07-18 23:10 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] mick crane <mick.crane@gmail.com> - 2020-07-18 23:30 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] <tomas@tuxteam.de> - 2020-07-19 10:40 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] mick crane <mick.crane@gmail.com> - 2020-07-20 13:20 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] David Christensen <dpchrist@holgerdanske.com> - 2020-07-19 01:50 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 12:50 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] David Christensen <dpchrist@holgerdanske.com> - 2020-07-19 23:30 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 13:40 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] William Michels <wjm1@caa.columbia.edu> - 2020-07-19 21:50 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] David Wright <deblis@lionunicorn.co.uk> - 2020-07-23 04:00 +0200
Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] songbird <songbird@anthive.com> - 2020-07-23 12:50 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 02:00 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-16 20:40 +0200
Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-11 17:00 +0200
Tex characters Teemu Likonen <tlikonen@iki.fi> - 2020-07-11 18:20 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 22:40 +0200
Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-13 14:00 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 07:50 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 18:10 +0200
Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-20 15:30 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-20 15:30 +0200
Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-20 18:20 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-10 12:50 +0200
Re: [offlist] posting style [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 05:00 +0200
Re: [offlist] posting style [was Re: Using .XCompose] Andrei POPESCU <andreimpopescu@gmail.com> - 2020-07-08 07:00 +0200
Re: [not-so-offlist] posting style [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-08 07:10 +0200
Re: [offlist] Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 05:00 +0200
Re: [offlist] Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-08 05:20 +0200
Re: [offlist] Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-08 05:40 +0200
Re: [offlist] Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-08 08:00 +0200
Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-07 00:40 +0200
tangent on man page locations [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-07 01:10 +0200
Re: tangent on man page locations [was Re: Using .XCompose] David Wright <deblis@lionunicorn.co.uk> - 2020-07-07 03:50 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-07 11:50 +0200
Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-07 13:40 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 16:30 +0200
Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-09 02:20 +0200
Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-09 04:20 +0200
Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-08 01:50 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-08 02:30 +0200
Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 05:10 +0200
Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 18:40 +0200
Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-09 04:00 +0200
Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-10 02:40 +0200
Re: Using .XCompose The Wanderer <wanderer@fastmail.fm> - 2020-07-10 02:50 +0200
Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-10 05:00 +0200
csiph-web