Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #224422 > unrolled thread

Using .XCompose

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-06 07:40 +0200
    Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-06 14:30 +0200
    Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-06 14:50 +0200
    Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-06 21:20 +0200
      Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-06 22:40 +0200
        Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-07 17:20 +0200
      Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-07 08:40 +0200
        Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-07 17:20 +0200
          Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-09 23:50 +0200
            Re: Using .XCompose [resend of xev_output_annotated.txt] davidson <davidson@freevolt.org> - 2020-07-10 03:40 +0200
            Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-10 10:50 +0200
              Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 10:40 +0200
                Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 12:30 +0200
                  Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-12 17:00 +0200
                    Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-13 06:00 +0200
                    Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-13 14:00 +0200
                      Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-13 20:30 +0200
                        Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-13 20:40 +0200
                          Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-13 20:50 +0200
                            Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-13 21:50 +0200
                              Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-14 05:40 +0200
                          Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-15 16:50 +0200
                          Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-15 17:10 +0200
                          Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-15 17:20 +0200
                    Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 01:30 +0200
                      Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 02:40 +0200
                      Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-16 20:40 +0200
                        how u mine 4 utf8 [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-18 03:20 +0200
                          UTF-8 Everywhere -- Re: how u mine 4 utf8 [was Re: Using .XCompose] Zenaan Harkness <zenaan@freedbms.net> - 2020-07-18 07:40 +0200
                          Re: how u mine 4 utf8 [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-18 09:20 +0200
                            Re: how u mine 4 utf8 [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 13:00 +0200
                          Fw: how u mine 4 utf8 [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 12:50 +0200
                            Re: Fw: how u mine 4 utf8 [was Re: Using .XCompose] Tom Browder <tom.browder@gmail.com> - 2020-07-19 13:10 +0200
                              Re: how u mine 4 utf8 [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 13:50 +0200
                        sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-18 12:10 +0200
                          Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Zenaan Harkness <zenaan@freedbms.net> - 2020-07-18 12:50 +0200
                            Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] William Michels <wjm1@caa.columbia.edu> - 2020-07-18 22:00 +0200
                          Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Andrei POPESCU <andreimpopescu@gmail.com> - 2020-07-18 13:20 +0200
                            Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] songbird <songbird@anthive.com> - 2020-07-18 21:00 +0200
                          Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] <tomas@tuxteam.de> - 2020-07-18 16:00 +0200
                            Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] mick crane <mick.crane@gmail.com> - 2020-07-18 22:40 +0200
                              Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] <tomas@tuxteam.de> - 2020-07-18 23:10 +0200
                                Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] mick crane <mick.crane@gmail.com> - 2020-07-18 23:30 +0200
                                  Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] <tomas@tuxteam.de> - 2020-07-19 10:40 +0200
                                    Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] mick crane <mick.crane@gmail.com> - 2020-07-20 13:20 +0200
                          Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] David Christensen <dpchrist@holgerdanske.com> - 2020-07-19 01:50 +0200
                            Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 12:50 +0200
                              Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] David Christensen <dpchrist@holgerdanske.com> - 2020-07-19 23:30 +0200
                          Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-19 13:40 +0200
                            Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] William Michels <wjm1@caa.columbia.edu> - 2020-07-19 21:50 +0200
                            Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] David Wright <deblis@lionunicorn.co.uk> - 2020-07-23 04:00 +0200
                              Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose] songbird <songbird@anthive.com> - 2020-07-23 12:50 +0200
                    Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 02:00 +0200
                      Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-16 20:40 +0200
                Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-11 17:00 +0200
                  Tex characters Teemu Likonen <tlikonen@iki.fi> - 2020-07-11 18:20 +0200
                  Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 22:40 +0200
                    Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-13 14:00 +0200
                      Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-16 07:50 +0200
                Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-11 18:10 +0200
                Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-20 15:30 +0200
                Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-20 15:30 +0200
                  Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-20 18:20 +0200
            Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-10 12:50 +0200
      Re: [offlist] posting style [was Re: Using .XCompose] Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 05:00 +0200
        Re: [offlist] posting style [was Re: Using .XCompose] Andrei POPESCU <andreimpopescu@gmail.com> - 2020-07-08 07:00 +0200
        Re: [not-so-offlist] posting style [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-08 07:10 +0200
      Re: [offlist] Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 05:00 +0200
        Re: [offlist] Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-08 05:20 +0200
        Re: [offlist] Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-08 05:40 +0200
          Re: [offlist] Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-08 08:00 +0200
    Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-07 00:40 +0200
      tangent on man page locations [was Re: Using .XCompose] davidson <davidson@freevolt.org> - 2020-07-07 01:10 +0200
        Re: tangent on man page locations [was Re: Using .XCompose] David Wright <deblis@lionunicorn.co.uk> - 2020-07-07 03:50 +0200
      Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-07 11:50 +0200
        Re: Using .XCompose Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-07 13:40 +0200
          Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 16:30 +0200
        Re: Using .XCompose David Wright <deblis@lionunicorn.co.uk> - 2020-07-09 02:20 +0200
          Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-09 04:20 +0200
    Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-08 01:50 +0200
      Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-08 02:30 +0200
      Fw: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 05:10 +0200
      Re: Using .XCompose Ajith R <ajithramayyan@yahoo.co.in> - 2020-07-08 18:40 +0200
        Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-09 04:00 +0200
          Re: Using .XCompose Zenaan Harkness <zenaan@freedbms.net> - 2020-07-10 02:40 +0200
            Re: Using .XCompose The Wanderer <wanderer@fastmail.fm> - 2020-07-10 02:50 +0200
        Re: Using .XCompose davidson <davidson@freevolt.org> - 2020-07-10 05:00 +0200

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

#224941 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

On 2020-07-18 14:56, tomas@tuxteam.de wrote:
> On Sat, Jul 18, 2020 at 09:59:46AM +0000, davidson wrote:
>> On Thu, 16 Jul 2020 Ajith R wrote:
>> >On Thursday 16 July 2020 4:54:09 AM IST davidson wrote:
>> 
>> [snip]
>> >>   $ sed 'y/\xc2\xa0/%/' somefile
>> >
>> >An off topic question: of sed, awk and perl, if I am to chose one to
>> >learn, which would you suggest.
> 
> Sed (and awk) are much lighter than perl. So if you find yourself
> peppering your shell scripts with lots of little calls to "that
> other program", you'll be better off with those two.
> 
> Perl will be useful if you are doing the heavy lifting in perl
> itself. Then it amortizes (I think this goes for both: computer
> run time and your brain time).
> 
> Cheers
> -- t
You see perl one liners and sed mentioned loads more than awk on the 
interweb.
Don't know why that is.
mick
-- 
Key ID    4BFEBB31

[toc] | [prev] | [next] | [standalone]

#224947 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jul 18, 2020 at 09:31:29PM +0100, mick crane wrote:

[...]

> You see perl one liners and sed mentioned loads more than awk on the
> interweb.
> Don't know why that is.

ISTR Perl had a bout of onelineritis back in its youth. This tends
to be somewhat detrimental to a language, because people tend to
believe this is the way you program.

Of course, hacking oneliners (with some measure) can be fun and
help in the language's mastery, but really doing serious work in
this way would be akin to writing a book on differential equations
in kōan form.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#224950 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

On 2020-07-18 22:08, tomas@tuxteam.de wrote:
> On Sat, Jul 18, 2020 at 09:31:29PM +0100, mick crane wrote:
> 
> [...]
> 
>> You see perl one liners and sed mentioned loads more than awk on the
>> interweb.
>> Don't know why that is.
> 
> ISTR Perl had a bout of onelineritis back in its youth. This tends
> to be somewhat detrimental to a language, because people tend to
> believe this is the way you program.
> 
> Of course, hacking oneliners (with some measure) can be fun and
> help in the language's mastery, but really doing serious work in
> this way would be akin to writing a book on differential equations
> in kōan form.
> 
> Cheers
> -- t
they do work tho' and can be astonishingly quick
cheers
mick

-- 
Key ID    4BFEBB31

[toc] | [prev] | [next] | [standalone]

#224959 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jul 18, 2020 at 10:20:00PM +0100, mick crane wrote:

[...]

> >Of course, hacking oneliners (with some measure) can be fun and
> >help in the language's mastery [...]

> they do work tho' and can be astonishingly quick

?

Quick in typing? In thinking? In compile time? In run time?

There's a common misconception that the less letters a program
has the faster it runs.

It's common, but it is a misconception ;-)

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#225032 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

On 2020-07-19 09:32, tomas@tuxteam.de wrote:
> On Sat, Jul 18, 2020 at 10:20:00PM +0100, mick crane wrote:
> 
> [...]
> 
>> >Of course, hacking oneliners (with some measure) can be fun and
>> >help in the language's mastery [...]
> 
>> they do work tho' and can be astonishingly quick
> 
> ?
> 
> Quick in typing? In thinking? In compile time? In run time?
> 
> There's a common misconception that the less letters a program
> has the faster it runs.
> 
> It's common, but it is a misconception ;-)
> 
run time, was the first time I saw a Perl oneliner somebody gave me.
Can't remember exactly, think was given a template HTML page and a 
directory of images make new HTML pages inserting image address from 
directory.
click done.

mick

-- 
Key ID    4BFEBB31

[toc] | [prev] | [next] | [standalone]

#224955 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

On 2020-07-18 02:59, davidson wrote:
> On Thu, 16 Jul 2020 Ajith R wrote:
>> On Thursday 16 July 2020 4:54:09 AM IST davidson wrote:
> 
> [snip]
>>>    $ sed 'y/\xc2\xa0/%/' somefile
>>
>> An off topic question: of sed, awk and perl, if I am to chose one to
>> learn, which would you suggest. I wanted to do some substitutions. I
>> read about them and decided on PERL because from what I understood,
>> it has better support for regular expressions and do almost
>> everything that sed and awk could do. Have I made the right
>> decision?

If your focus is regular expressions, then you should read "Mastering 
Regular Expressions", 3rd Edition:

https://www.oreilly.com/library/view/mastering-regular-expressions/0596528124/


The author compares and contrasts regular expression capabilities in 
various tools.  I seem to recall that he puts Perl at the top of the 
heap, and notes that Perl compatible regular expressions (PCRE) are 
available via libraries in other programming languages.


I will also say that I have used Perl for 20+ years, and Perl has 
allowed me to expand my knowledge and skills tremendously.  I have 
accomplished a lot of useful work with Perl.


As the Perl slogan goes, "There is more than one way to do it".  Perl 
delivers.


Another slogan starts with "Perl makes easy things easy", but I have my 
doubts about the second half "makes hard things possible".


20 years ago, Moore's Law ruled.  You bought a new Pentium computer 
every year and a half, and your Perl script ran faster.


Today, Amdahl's Law rules.  You buy a new Core or Ryzen chip every year 
and a half and your Perl script runs at the same speed.


Perl is fundamentally a sequential programming language.  Concurrent 
programming was bolted on with a library, and that library has a huge 
disclaimer:

     https://metacpan.org/pod/threads

     "WARNING

     The "interpreter-based threads" provided by Perl are not the fast,
     lightweight system for multitasking that one might expect or hope
     for. Threads are implemented in a way that make them easy to misuse.
     Few people know how to use them correctly or will be able to provide
     help.

     The use of interpreter-based threads in perl is officially
     discouraged."


Being the lazy, impatient, and over-proud Perl programmer that I am, I 
disregarded the above and have been working on a Perl library for 
multi-threaded flow-based programming.  OMG.  I don't know if I can or 
will succeed.  It is the most difficult Perl code I have ever worked on.


I have experimented with Go in the past.  I plan to start learning 
Erlang shortly.


David

[toc] | [prev] | [next] | [standalone]

#224963 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

Hi,

> I seem to recall that he puts Perl at the top of the 
> heap, and notes that Perl compatible regular expressions (PCRE) are 
> available via libraries in other programming languages.

Thanks for confirming that I didn't make a wrong choice. Programs that claim to use PCRE don't support everything that PERL does.

I wanted to clean many documents (Wikipedia dump) to analyse the Malayalam content. As I was not comfortable with scripting, I was looking for some prorgam that could remove the foreign language text from the files. As, I could find none that could do the job, I had to use a Perl script with the line below (among others)

s/[^\p{Block: Malayalam}\p{Block: Basic_Latin}\p{Block: General_Punctuation}\s]//g; # remove characters outside the specified unicode blocks.

As of now, the simple substitute command of perl is sufficient for my requirements. Even that one command appears powerful compared to others.

ajith

[toc] | [prev] | [next] | [standalone]

#225015 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

One last point I missed in my previous post -- big files.  Many Perl 
functions and/or libraries expect to do everything in RAM.  This becomes 
a problem when I want to compress, encrypt, save, and checksum 14 GiB 
system drive images using a live drive and a Perl program in computers 
without large amounts of memory (64 GiB?).  So, I ended up writing a 
Perl program that is truly a script -- it formulates and invokes 
gzip(1), ccrypt(1), md5sum(1), and sha256sum(1) commands as required. 
Traditional Bourne shell scripts are ideally suited for such tasks, and 
the syntax can be cleaner than Perl in simple cases.


On 2020-07-19 03:46, Ajith R wrote:

>> I seem to recall that he puts Perl at the top of the
>> heap, and notes that Perl compatible regular expressions (PCRE) are
>> available via libraries in other programming languages.
> 
> Thanks for confirming that I didn't make a wrong choice. Programs that claim to use PCRE don't support everything that PERL does.

My use of regular expressions is primarily via grep/ egrep and Perl.  I 
am curious to see what Erlang offers.


> I wanted to clean many documents (Wikipedia dump) to analyse the Malayalam content. As I was not comfortable with scripting, I was looking for some prorgam that could remove the foreign language text from the files. As, I could find none that could do the job, I had to use a Perl script with the line below (among others)
> 
> s/[^\p{Block: Malayalam}\p{Block: Basic_Latin}\p{Block: General_Punctuation}\s]//g; # remove characters outside the specified unicode blocks.
> 
> As of now, the simple substitute command of perl is sufficient for my requirements. Even that one command appears powerful compared to others.

That sounds like you need lexing and parsing, followed by your desired 
processing.  In simple cases, Perl regular expressions can accomplish 
two or three of those tasks.  But as complexity grows, you will need 
more and more code.  Writing a lexer/ parser in any language is a 
non-trivial task.  I have used the Perl 'LWP' library to parse HTML 4 
pages, but I have not tried to parse HTML 5 and/or Wikipedia pages.  I 
would look for a library:

https://metacpan.org/search?q=parse

https://metacpan.org/search?q=wikipedia


I spent a little time with Raku (formerly Perl 6).  AIUI improving 
regular expressions was a design goal of Perl 6, and features were added 
specifically for parsing.  There is a book dedicated to the subject.  I 
have a friend who put some time into this area, and he seemed impressed:

https://www.apress.com/us/book/9781484232279


David

[toc] | [prev] | [next] | [standalone]

#224968 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

Hi,

> First, there is a somewhat specific question about unspecified
> substitutions. For all I know about these substitutions, you might
> actually need XSLT to do them properly.

The substitution that I had in mind requires referring to characters based on their unicode properties like script, block...

> I think you should absolutely use perl if it makes you happy.
> Unix has a pretty interesting collection of various small tools (which
> "do one thing and do it well" as you may hear), and shells facilitate
> hooking up their outputs and inputs. Almost as if they were made to do
> just that.

I don't prescribe to using a tool for the sake of happiness. With my limited knoweldege I want to select one that is adequate to do the job.
The subsitution that I wanted in many text files was deleting text from languages other than Malayalam,english and punctuation. This required a program that could match charcters based on their unicode character of block / script. I didn't find anything to suggest that sed could do that. May be, I didn't search properly.
Did  I miss a utility(including sed) that can do the kind of substitution I mentioned above?

Thanks,
ajith

[toc] | [prev] | [next] | [standalone]

#225010 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

Hello Ajith,

Tom Browder suggests taking a look at Raku (née Perl6), and I concur.
While I don't know Malayalam at all, I can write the regex code below
with ease:

> #all code below using the Raku REPL:
> say '0123456789'.chars;
10
> say $/ if '0123456789' ~~ /  \d+ /;
｢0123456789｣

> #now with Bengali digits:
> say '০১২৩৪৫৬৭৮৯'.chars;
10
> say $/ if '০১২৩৪৫৬৭৮৯' ~~ /  \d+ /;
｢০১২৩৪৫৬৭৮৯｣

>#now with Malayalam digits:
> say '൦൧൨൩൪൫൬൭൮൯'.chars;
10
> say $/ if '൦൧൨൩൪൫൬൭൮൯' ~~ /  \d+ /;
｢൦൧൨൩൪൫൬൭൮൯｣


More info here:

https://www.nntp.perl.org/group/perl.perl6.users/2020/06/msg8828.html
https://www.nntp.perl.org/group/perl.perl6.users/2020/06/msg8845.html

HTH, Bill.



On Sun, Jul 19, 2020 at 4:36 AM Ajith R <ajithramayyan@yahoo.co.in> wrote:
>
> Hi,
>
> > First, there is a somewhat specific question about unspecified
> > substitutions. For all I know about these substitutions, you might
> > actually need XSLT to do them properly.
>
> The substitution that I had in mind requires referring to characters based on their unicode properties like script, block...
>
> > I think you should absolutely use perl if it makes you happy.
> > Unix has a pretty interesting collection of various small tools (which
> > "do one thing and do it well" as you may hear), and shells facilitate
> > hooking up their outputs and inputs. Almost as if they were made to do
> > just that.
>
> I don't prescribe to using a tool for the sake of happiness. With my limited knoweldege I want to select one that is adequate to do the job.
> The subsitution that I wanted in many text files was deleting text from languages other than Malayalam,english and punctuation. This required a program that could match charcters based on their unicode character of block / script. I didn't find anything to suggest that sed could do that. May be, I didn't search properly.
> Did  I miss a utility(including sed) that can do the kind of substitution I mentioned above?
>
> Thanks,
> ajith
>

[toc] | [prev] | [next] | [standalone]

#225146 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

On Sun 19 Jul 2020 at 11:35:31 (+0000), Ajith R wrote:
> 
> > First, there is a somewhat specific question about unspecified
> > substitutions. For all I know about these substitutions, you might
> > actually need XSLT to do them properly.
> 
> The substitution that I had in mind requires referring to characters based on their unicode properties like script, block...
> 
> > I think you should absolutely use perl if it makes you happy.
> > Unix has a pretty interesting collection of various small tools (which
> > "do one thing and do it well" as you may hear), and shells facilitate
> > hooking up their outputs and inputs. Almost as if they were made to do
> > just that.
> 
> I don't prescribe to using a tool for the sake of happiness. With my limited knoweldege I want to select one that is adequate to do the job.
> The subsitution that I wanted in many text files was deleting text from languages other than Malayalam,english and punctuation. This required a program that could match charcters based on their unicode character of block / script. I didn't find anything to suggest that sed could do that. May be, I didn't search properly.
> Did  I miss a utility(including sed) that can do the kind of substitution I mentioned above?

I've always found exploring new languages very invigorating: the
excitement of having new abilities (the modern generation might
call them new superpowers), and the pleasure of learning itself.
I find it rather a shame that enjoyment isn't at least an important
criterion in selecting your programming tools.

One of the most pleasurable times in my career was when our Research
Computing Advisor thrust a copy of the Green Book into my hands.
http://www.math.bas.bg/bantchev/place/snobol/gpp-2ed.pdf
After years of Fortran, this was my first experience of a designed
computer language, with dynamic allocation of memory, dynamic typing,
associative arrays, and its claim to fame, pattern matching.

With over 40 years of language development since then, it's probably
difficult for anyone now to get quite the same buzz from meeting this
language. (BTW I was never a user of SNOBOL4 itself, but the compiled
version, SPITBOL, which was much faster, and available to me on IBM
OS/360 and its successors.) Its pattern matching was very ahead of
its time.

I had to move to Perl4 for two reasons: 1) the flight from mainframes
in academia, in favour of Unix and PCs, meant that Spitbol became
inaccessible for me; 2) Perl, which was developed a few years earlier
than Python, had a reasonable DOS/Extended Memory implementation
available at the time I got my first PC, a 386DX. Perl gave me an
approximation to a Unix shell and its utilities, and the associative
arrays that I required, but its pattern matching was much more limited
than Spitbol's.

By the time Debian was released, I'd been introduced to Python, and
decided that it looked more attractive than making the transition
Perl4→5. I've never regretted it, despite the hiatuses of the 2→3
and Unicode transitions, particularly as Perl seems to be standing
at a fork in the road so much of the time, whereas Python's behaviour
appears to have been more decisive.

As for your choosing which language to learn, I would spend your time
and effort on a language like Perl or Python. You'll probably pick up
utilities like grep and sed with the greatest of ease if you're
familiar with P⁓.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#225163 — Re: sed:awk:perl::rock:paper:chainsaw [was Re: Using .XCompose]

David Wright wrote:
...
> One of the most pleasurable times in my career was when our Research
> Computing Advisor thrust a copy of the Green Book into my hands.
> http://www.math.bas.bg/bantchev/place/snobol/gpp-2ed.pdf
> After years of Fortran, this was my first experience of a designed
> computer language, with dynamic allocation of memory, dynamic typing,
> associative arrays, and its claim to fame, pattern matching.
>
> With over 40 years of language development since then, it's probably
> difficult for anyone now to get quite the same buzz from meeting this
> language. (BTW I was never a user of SNOBOL4 itself, but the compiled
> version, SPITBOL, which was much faster, and available to me on IBM
> OS/360 and its successors.) Its pattern matching was very ahead of
> its time.

...

  yes that was one of languages we surveyed in 
the computer language class.  :)  much fun back 
then.  same as lisp, algol, ada, smalltalk, etc.

  i think i still have the green book here 
someplace.

  time marches on and i've mostly retired from 
coding unless i can come up with something that
catches my interest strongly enough to keep me
busy through a winter.  summertime i'm too busy
gardening to put much time into anything 
significant.


  songbird

[toc] | [prev] | [next] | [standalone]

#224866

[Multipart message — attachments visible in raw view] — view raw

On Sun, 12 Jul 2020 Ajith R wrote:

> Could you please see if any of the lines in this message too behaves
> similarly

ie, see whether there are nonbreaking spaces in the message body.

> ?

The message body --of the message to which I reply-- I saved in the
file ajith_msgbody.txt, which features in the commands below.

   $ grep '%' ajith_msgbody.txt # verify '%' character absent from original
   $ echo $? # exit status of above command; 1 indicates no match found
   1
   $ sed 'y/\xc2\xa0/%/' ajith_msgbody.txt >ajith_msgbody_replace_nbsp.txt
   $

I attach ajith_msgbody_replace_nbsp.txt, which contains a '%'
character wherever a nonbreaking space was in the original.

-- 
Ce qui est important est rarement urgent
et ce qui est urgent est rarement important
-- Dwight David Eisenhower

[toc] | [prev] | [next] | [standalone]

#224884

 On Thursday 16 July 2020 5:27:16 AM IST davidson wrote:




> ie, see whether there are nonbreaking spaces in the message body.

> I attach ajith_msgbody_replace_nbsp.txt, which contains a '%'

> character wherever a nonbreaking space was in the original.




Thanks. So, the non-breaking spaces are getting added in the email system.

[toc] | [prev] | [next] | [standalone]

#224701

On Sat 11 Jul 2020 at 08:39:26 (+0000), davidson wrote:
> On Fri, 10 Jul 2020 Ajith R wrote:
> > Thanks for your experiment. The fact that it works atleast in some
> > applications gives hope.
> 
> […]
> 
> Curious parties (like myself) wishing to understand the orthographic
> peculiarities at issue here, might find the following section of the
> Wikipedia entry on Malayalam script provides brief, helpful
> background:
> 
>  https://en.wikipedia.org/wiki/Malayalam_script#Ligatures

Naturally, this page concerns itself mainly with how "letters" are
written to compose words on the page. But there's a short section at
the end about how the "letters" are represented in the computer and,
possibly, how they are typed in in order to achieve that internal
representation.

Again, I can only talk about languages I understand. In English,
there's a convention that fi is printed differently from the
letters f and i. Similarly, ffi, fl and ffl. However, this
modification is performed on output, in accordance with the font
being used. "Difficult" is always typed in and stored with the
letters f f i.

OTOH, although in the early days of computing, the lack of different
characters forced people to use 1/2 to represent a half, there's no
doubt that ½ exists as a character which can be input and stored as
just that: ½. My father would think it odd that computer keyboards
(often/usually) don't have that key, because all the typewriters
and adding machines from his era did.

So it's natural that I should wish to define a sequence of keystrokes
that types ½, and that's what I have. It's CapsLock 1 2. It might not
work absolutely everywhere, but typically they're going to be places
I wouldn't want or need to use it. For example, the getty where I log
in (can't), and anywhere where keys aren't reflected, like
passwords/phrases (wouldn't).

So those examples, fi and ½, illustrate the difference between
modification on output and input in English. I can't judge how the OP
views this, nor whether they are contravening some conventions in
their own computer culture by trying to make their changes. This
doesn't even address how a computer responds to a command line
written in non-latin script.

> The "appropriate geminate glyph" refered to in property (2) above is
> illustrated in this table, in the "ligated" row under the column
> headed "ṅṅa".
> 
>  https://en.wikipedia.org/wiki/Malayalam_script#Common_consonant_ligatures
> 
> > I copied your exact sentences to my compose file. 
> > 
> > In Konsole the command cat ~/.XCompose gives the result:
> > 
> > include "%L" 
> > <U0D19>  : "ങ്ങ" # Ajith's auto-geminate rule, (U0D19) => (U0D19) (U0D4D) (U0D19)
> > <Multi_key> <s> <x> : "✄"  # (U2704) white scissors, h/t David Wright
> > <Z>                : "ARGA WARGA IN THE DARGA instead of Z"
> 
> Do you know what I see in four lines above, representing the content
> of your ~/.XCompose file?  I see "non-breaking space" characters
> (two-byte sequences 0xc2a0). I see 8 of them in the last line alone
> (the <Z> rule). Do you know what non-breaking space characters are?
> 
> I'm not quite sure how best to describe them, myself.

In TeX, you would type in Mr~Wright, because it's considered that Mr
Wright is not the polite way to print a name in a book. I've never
tested whether modern versions of text will handle Mr Wright directly
as input, because using ~ explicitly has the advantage of making the
modification more apparent. Otherwise, they're indistinguishable from
spaces in text and on the command line, only showing in emacs, where
they're coloured like all "idiosyncratic" whitespace.

> But I will tell you what they *aren't*. They aren't spaces. They
> aren't tabs. And the only thing their presence is going to do --in
> your code and in your configuration files-- is wreak havoc.

Similarly, a search like   /Mr Wright   will no longer work in less;
you might have to search   /Mr.Wright   instead, with a wildcard.

> Their name is very ironic. For our purposes,
> 
>   1. they aren't spaces, and
>   2. they break EVERYTHING.
> 
> Whatever it is you are doing that puts them there, STOP DOING IT.
> 
> […]
> 
> > So, as Zeenan says, there is something fundamentally wrong in my
> > system. I have to find that and correct it or reinstall everything.

I would attempt to back out of the changes made. Removing/hiding
.XCompose is an obvious first step. The keymap changes might be
trickier: are there changes in /usr/share/X11/kbd/ or have they
only added files in /etc/X11/kbd/? I know knothing about DE menus.
It would appear from others' posts that DEs can change anything
and everything.

> > > firefox-esr issued the following errors, which appear to be explanatory:
> > > (firefox-esr:11619): Gtk-WARNING **: 16:27:05.630: GTK+ supports to output one char only:

I've never tried to do anything about special characters with FF.
They either work (like ½) in both the address and search bars, or
I expect google or whatever to sort it out like typos (cf role
and rôle). But then, I don't write emails in a browser, or anything
like that.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#224703 — Tex characters

[Multipart message — attachments visible in raw view] — view raw

David Wright [2020-07-11T09:51:07-05] wrote:

> In TeX, you would type in Mr~Wright, because it's considered that Mr
> Wright is not the polite way to print a name in a book. I've never
> tested whether modern versions of text will handle Mr Wright directly
> as input, [...]

At least with Xetex and Luatex compilers both tilde (~) and U+00A0
NO-BREAK SPACE can be used as a non-breaking space. There is one
difference. Tilde is a stretchable space just like normal word space.
U+00A0 is a fixed-width space and more suitable for keeping some closely
related parts together. (And let's not forget \, command which is a
fixed 1/6 em space.)

-- 
/// Teemu Likonen - .-.. http://www.iki.fi/tlikonen/
// OpenPGP: 4E1055DC84E9DFF613D78557719D69D324539450

[toc] | [prev] | [next] | [standalone]

#224704

[Multipart message — attachments visible in raw view] — view raw

On Sat, 11 Jul 2020 David Wright wrote:
> I would attempt to back out of the changes made. Removing/hiding
> .XCompose is an obvious first step.

Yes.

'!' marks the spot of nonbreaking spaces that made it into OP's first
report of odd behavior, upon testing the white scissors XCompose rule:

   $ grep "WHITE SCISSORS"  d-u_xcompose_2020-07-08.nbsp | tr $'\xc2\xa0' \!
   <Multi_key> <s> <x>!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! : "✄"!!!! U2704 # WHITE SCISSORS

This message:

  https://lists.debian.org/msgid-search/620563117.3978825.1594226239402@mail.yahoo.com
   "Now, things are even more strange. When I press the Caps Lock, a
   strange character appears. [...]"

-- 
Ce qui est important est rarement urgent
et ce qui est urgent est rarement important

                --Dwight David Eisenhower

[toc] | [prev] | [next] | [standalone]

#224737

On Sat, Jul 11, 2020 at 08:32:34PM +0000, davidson wrote:
> '!' marks the spot of nonbreaking spaces that made it into OP's first
> report of odd behavior, upon testing the white scissors XCompose rule:
> 
>   $ grep "WHITE SCISSORS"  d-u_xcompose_2020-07-08.nbsp | tr $'\xc2\xa0' \!
>   <Multi_key> <s> <x>!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! : "✄"!!!! U2704 # WHITE SCISSORS

Note that tr does not handle multi-character sequences.  If you pass
something like

tr abc xyz

It does *not* look for "abc" sequences and convert only those sequences.
Rather, it looks at single characters.  It converts 'a' to 'x', and 'b'
to 'y', and 'c' to 'z'.

The number of characters in the first pattern is supposed to match the
number of characters in the second pattern, so that there is a 1:1
mapping.

GNU tr also does not handle multi-byte *characters* correctly (which
violates POSIX -- it's a known bug).

So, your tr command actually converts all c2 bytes into ! and all a0
bytes into ! as well.  Not *just* c2a0 pairs.

Nevertheless, this is useful as a first pass approximation to say that,
hey, there *might* be a bunch of NBSPs here, and you should take a
closer look.

NBSPs most often result when someone gets lazy and pastes a line from
a web page or from a Microsoft Word/Excel document into a Unix
terminal or X11 application, instead of pasting just the characters
they actually want.  Web pages, especially *older* web pages, often
use NBSPs for primitive formatting.

[toc] | [prev] | [next] | [standalone]

#224869

[Multipart message — attachments visible in raw view] — view raw

On Mon, 13 Jul 2020 Greg Wooledge wrote:
> On Sat, Jul 11, 2020 at 08:32:34PM +0000, davidson wrote:
>> '!' marks the spot of nonbreaking spaces that made it into OP's first
>> report of odd behavior, upon testing the white scissors XCompose rule:
>>
>>   $ grep "WHITE SCISSORS"  d-u_xcompose_2020-07-08.nbsp | tr $'\xc2\xa0' \!
>>   <Multi_key> <s> <x>!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! : "✄"!!!! U2704 # WHITE SCISSORS
>
> Note that tr does not handle multi-character sequences.  If you pass
> something like
>
> tr abc xyz
>
> It does *not* look for "abc" sequences and convert only those sequences.
> Rather, it looks at single characters.  It converts 'a' to 'x', and 'b'
> to 'y', and 'c' to 'z'.
>
> The number of characters in the first pattern is supposed to match the
> number of characters in the second pattern, so that there is a 1:1
> mapping.

This much, as it happens, I knew.

My intent was to make plain the nonbreaking spaces for the list, and
in particular for the list archives. (Because it appears that the
versions of messages posted in the web-archives do not preserve such
characters.)

> GNU tr also does not handle multi-byte *characters* correctly (which
> violates POSIX -- it's a known bug).

And *this* I did *not* know. It was my incorrect belief that the two
bytes of tr's first argument would be treated by tr as a single
character.

It wasn't a *firm* belief. I just did not think it through
carefully. Had I wanted it's actual behavior instead, I may have even
expected it. Stopped clock is correct once in a while!

I see now that "info tr" mentions this behavior up-front. (And
indicates that in future tr will support multi-byte characters.)

In the man page tr(1) I see nothing about all this. It simply talks
about "characters", and assumes the reader is some kind of K&R
obsessed mind-reader who knows "Well, of course a 'character' here
just denotes an octet of bits."

> So, your tr command actually converts all c2 bytes into ! and all a0
> bytes into ! as well.  Not *just* c2a0 pairs.

Thank you for catching all this, and for concise and comprehensive
explanations.

tr's output line was significantly longer than the line it received as
input. I ought to have noticed.

> Nevertheless, this is useful as a first pass approximation to say that,
> hey, there *might* be a bunch of NBSPs here, and you should take a
> closer look.

You are being charitable.

Since it does not (currently) know what characters are when they
aren't composed of single octets, it was the wrong tool for the job.

   sed 'y/\xc2\xa0/!/'

seems to do the right thing.

> NBSPs most often result when someone gets lazy and pastes a line
> from a web page or from a Microsoft Word/Excel document into a Unix
> terminal or X11 application, instead of pasting just the characters
> they actually want.  Web pages, especially *older* web pages, often
> use NBSPs for primitive formatting.

Noted.

-- 
Ce qui est important est rarement urgent
et ce qui est urgent est rarement important
-- Dwight David Eisenhower

[toc] | [prev] | [next] | [standalone]

#224702

On Sat, 11 Jul 2020 davidson wrote:
> On Fri, 10 Jul 2020 Ajith R wrote:

[snip]

>> So, as Zeenan says, there is something fundamentally wrong in my
>> system. I have to find that and correct it or reinstall everything.
>
> Is the situation still as you describe here?
>
>  8 July 2020 Ajith R to debian-user
>  https://lists.debian.org/msgid-search/620563117.3978825.1594226239402@mail.yahoo.com

[snip]

> If this is still so, we ought to figure out what's going on. To this
> end, I previously asked for the output of
>
> $ xmodmap
>
> and
>
> $ xmodmap -pk

These are, by the way, purely diagnostic command lines. They are
requests to print information only. If you issue them as written
above, they will alter nothing except (possibly) our state of
ignorance.

That is, in my ignorant optimism I imagine their output might help
somebody make informed, helpful suggestions.

While I'm here, I will add to that pair a request for the output from
a third, also purely diagnostic:

  $ setxkbmap -print

-- 
Ce qui est important est rarement urgent
et ce qui est urgent est rarement important

                --Dwight David Eisenhower

[toc] | [prev] | [next] | [standalone]

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

Back to top | Article view | linux.debian.user

csiph-web