Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110978 > unrolled thread

Well, I finally ran into a Python Unicode problem, sort of

Back to article view | Back to comp.lang.python

Contents

  Well, I finally ran into a Python Unicode problem, sort of John Ladasky <john_ladasky@sbcglobal.net> - 2016-07-02 23:58 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 00:07 -0700
      Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 10:26 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 01:38 -0700
          Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 12:01 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of Chris Angelico <rosuav@gmail.com> - 2016-07-04 01:14 +1000
            Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 14:21 -0700
      Re: Well, I finally ran into a Python Unicode problem, sort of John Ladasky <john_ladasky@sbcglobal.net> - 2016-07-03 09:24 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of Rustom Mody <rustompmody@gmail.com> - 2016-07-03 00:26 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-03 10:29 +0300
      Re: Well, I finally ran into a Python Unicode problem, sort of Robert Kern <robert.kern@gmail.com> - 2016-07-03 10:16 +0100
    Re: Well, I finally ran into a Python Unicode problem, sort of Chris Angelico <rosuav@gmail.com> - 2016-07-03 17:41 +1000
      Re: Well, I finally ran into a Python Unicode problem, sort of John Ladasky <john_ladasky@sbcglobal.net> - 2016-07-03 11:39 -0700
        Re: Well, I finally ran into a Python Unicode problem, sort of MRAB <python@mrabarnett.plus.com> - 2016-07-03 20:41 +0100
        Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 14:28 -0700
          Re: Well, I finally ran into a Python Unicode problem, sort of Steven D'Aprano <steve@pearwood.info> - 2016-07-04 12:33 +1000
            Re: Well, I finally ran into a Python Unicode problem, sort of Rustom Mody <rustompmody@gmail.com> - 2016-07-03 22:14 -0700
              Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-04 01:11 -0700
                Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-04 13:10 +0300
                Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-04 12:04 +0300
              Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-04 09:08 +0300
    Re: Well, I finally ran into a Python Unicode problem, sort of Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2016-07-03 11:53 +0200
      Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 14:01 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of Christian Gollwitzer <auriocus@gmx.de> - 2016-07-03 13:16 +0200
          Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 14:22 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of Christian Gollwitzer <auriocus@gmx.de> - 2016-07-03 16:05 +0200
              Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 17:39 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of Random832 <random832@fastmail.com> - 2016-07-03 16:49 -0400
              Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-04 00:25 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-03 12:50 +0100
          Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 14:26 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of eryk sun <eryksun@gmail.com> - 2016-07-03 23:46 +0000
      Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 17:00 -0700
        Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 01:17 +0100
          Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 17:24 -0700
            Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 01:39 +0100
              Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 18:15 -0700
                Re: Well, I finally ran into a Python Unicode problem, sort of Random832 <random832@fastmail.com> - 2016-07-03 22:20 -0400
                Re: Well, I finally ran into a Python Unicode problem, sort of Rustom Mody <rustompmody@gmail.com> - 2016-07-04 05:16 -0700
                Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 11:26 +0100
          Re: Well, I finally ran into a Python Unicode problem, sort of Steven D'Aprano <steve@pearwood.info> - 2016-07-04 12:30 +1000
            Re: Well, I finally ran into a Python Unicode problem, sort of Ned Batchelder <ned@nedbatchelder.com> - 2016-07-04 05:47 -0700
              Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 15:36 +0100
                Re: Well, I finally ran into a Python Unicode problem, sort of Ned Batchelder <ned@nedbatchelder.com> - 2016-07-04 07:46 -0700
                  Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 17:16 +0100
                    Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-04 19:44 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 11:05 +0100
              Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-04 13:26 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of Random832 <random832@fastmail.com> - 2016-07-03 22:16 -0400

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#111070

Rustom Mody writes:

> Subscripts OTOH as part of identifier-lexemes doesn't seem to have any
> issues

They have the general issue that one might *want* them interpreted as
indexes, so that a₁ would mean the same as a[1].

Mathematical symbols face similar issues. One would not *want* them all
be binary operators; a specific level of precedence would not be good
for all uses; and some uses of some symbols need chaining and then
parentheses do not help. Just for the starters.

> My main point being unicode gives a wide repertory -- thats good
> It also gives char-classification -- thats a start
> But its not enough for designing a (modern) programming

So I agree. Something could be done, but if the intention is to allow
mathematical notation, it needs to be done with care.

(And no, I'm not saying Python needs to do anything at this time, and I
do not express any opinion on how likely Python is to do anything about
Unicode math at this time or ever, and so on. Just that I would not be
happy to have all those symbols available in a way that is not usable
for the intended purpose so please do take care.)

[toc] | [prev] | [next] | [standalone]

#110987

John Ladasky <john_ladasky@sbcglobal.net> writes:

> from math import pi as π
> [...]
> c = 2 * π * r

> Up until today, every character I've tried has been accepted by the
> Python interpreter as a legitimate character for inclusion in a
> variable name. Now I'm copying a formula which defines a gradient. The
> nabla symbol (∇) is used in the naming of gradients. Python isn't
> having it. The interpreter throws a "SyntaxError: invalid character in
> identifier" when it encounters the ∇.

The rules are at
https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers

To me it makes a lot of sense to *not* include category Sm characters in
identifiers, since they are usually used to denote operators (like +).
It would be very confusing to have a variable named ∇f, as confusing as
naming a variable a+b or √x.

-- Alain.

[toc] | [prev] | [next] | [standalone]

#110988

Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:

> It would be very confusing to have a variable named ∇f, as confusing
> as naming a variable a+b or √x.

Scheme allows *any* characters whatsoever in identifiers.


Marko

[toc] | [prev] | [next] | [standalone]

#110989

Am 03.07.16 um 13:01 schrieb Marko Rauhamaa:
> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
>
>> It would be very confusing to have a variable named ∇f, as confusing
>> as naming a variable a+b or √x.
>
> Scheme allows *any* characters whatsoever in identifiers.

Parentheses?

	Christian

[toc] | [prev] | [next] | [standalone]

#110990

Christian Gollwitzer <auriocus@gmx.de>:

> Am 03.07.16 um 13:01 schrieb Marko Rauhamaa:
>> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
>>
>>> It would be very confusing to have a variable named ∇f, as confusing
>>> as naming a variable a+b or √x.
>>
>> Scheme allows *any* characters whatsoever in identifiers.
>
> Parentheses?

Yes.

Hint: Python allows *any* characters whatsoever in strings.


Marko

[toc] | [prev] | [next] | [standalone]

#110995

Am 03.07.16 um 13:22 schrieb Marko Rauhamaa:
> Christian Gollwitzer <auriocus@gmx.de>:
>
>> Am 03.07.16 um 13:01 schrieb Marko Rauhamaa:
>>> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
>>>
>>>> It would be very confusing to have a variable named ∇f, as confusing
>>>> as naming a variable a+b or √x.
>>>
>>> Scheme allows *any* characters whatsoever in identifiers.
>>
>> Parentheses?
>
> Yes.
>
> Hint: Python allows *any* characters whatsoever in strings.

My knowledge of Scheme is rusty. How do you do that? Consider

	(define x 'hello)

then the x is the identifier, isn't it? How can you include a 
metacharacter like space, ', or ( in it? I'm using 
https://repl.it/languages/scheme to try it out.

Another language which allows any characters in identifiers is Tcl. Here 
you can quote identifiers:

	set {a b} c

creates a variable "a b" with a space in it, because there is no 
distinction between quoted/unquoted. Metacharacters can be included by 
\-escapes. How does that work in Scheme?

	Christian

[toc] | [prev] | [next] | [standalone]

#110996

Christian Gollwitzer <auriocus@gmx.de>:

> Am 03.07.16 um 13:22 schrieb Marko Rauhamaa:
>> Christian Gollwitzer <auriocus@gmx.de>:
>>> Am 03.07.16 um 13:01 schrieb Marko Rauhamaa:
>>>> Scheme allows *any* characters whatsoever in identifiers.
>>> Parentheses?
>> Yes.
>
> My knowledge of Scheme is rusty. How do you do that?

   Moreover, all characters whose Unicode scalar values are greater than
   127 and whose Unicode category is Lu, Ll, Lt, Lm, Lo, Mn, Mc, Me, Nd,
   Nl, No, Pd, Pc, Po, Sc, Sm, Sk, So, or Co can be used within
   identifiers. In addition, any character can be used within an
   identifier when specified via an <inline hex escape>. For example,
   the identifier H\x65;llo is the same as the identifier Hello, and the
   identifier \x3BB; is the same as the identifier λ.
   <URL: http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-7.html#node_sec_4.2.4>

Guile doesn't support the R6RS inline hex escape notation. Instead, it
natively supports a notation of its own:

   #{foo bar}#

   #{what
   ever}#

   #{4242}#

Or the R7RS notation:

   |foo bar|
   |\x3BB; is a greek lambda|
   |\| is a vertical bar|

   <URL: https://www.gnu.org/software/guile/manual/html_node/Symbol-Rea
   d-Syntax.html#index-r7rs_002dsymbols>


Marko

[toc] | [prev] | [next] | [standalone]

#111009

On Sun, Jul 3, 2016, at 07:22, Marko Rauhamaa wrote:
> Christian Gollwitzer <auriocus@gmx.de>:
> > Am 03.07.16 um 13:01 schrieb Marko Rauhamaa:
> >> Scheme allows *any* characters whatsoever in identifiers.
> >
> > Parentheses?
> 
> Yes.
> 
> Hint: Python allows *any* characters whatsoever in strings.

Being able to put any character in a symbol doesn't make those strings
identifiers, any more than passing them to getattr/setattr (or
__import__, something's __name__, etc) does in Python.

[toc] | [prev] | [next] | [standalone]

#111012

Random832 <random832@fastmail.com>:

> Being able to put any character in a symbol doesn't make those strings
> identifiers, any more than passing them to getattr/setattr (or
> __import__, something's __name__, etc) does in Python.

From R7RS, the newest Scheme standard (p. 61-62):

     7.1.1. Lexical structure
     [...]
     〈vertical line〉 → |
     [...]
     〈identifier〉 → 〈initial〉 〈subsequent〉*
          | 〈vertical line〉 〈symbol element〉* 〈vertical line〉
          | 〈peculiar identifier〉
     〈initial〉 → 〈letter〉 | 〈special initial〉
     〈letter〉 → a | b | c | ... | z
         | A | B | C | ... | Z
     〈special initial〉 → ! | $ | % | & | * | / | : | < | =
         | > | ? | ^ | _ | ~
     〈subsequent〉 → 〈initial〉 | 〈digit〉
         | 〈special subsequent〉
     〈digit〉 → 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
     〈hex digit〉 → 〈digit〉 | a | b | c | d | e | f
     〈explicit sign〉 → + | -
     〈special subsequent〉 → 〈explicit sign〉 | . | @
     〈inline hex escape〉 → \x〈hex scalar value〉;
     〈hex scalar value〉 → 〈hex digit〉 +
     〈mnemonic escape〉 → \a | \b | \t | \n | \r
     〈peculiar identifier〉 → 〈explicit sign〉
         | 〈explicit sign〉 〈sign subsequent〉 〈subsequent〉*
         | 〈explicit sign〉 . 〈dot subsequent〉 〈subsequent〉*
         | . 〈dot subsequent〉 〈subsequent〉*
     〈dot subsequent〉 → 〈sign subsequent〉 | .
     〈sign subsequent〉 → 〈initial〉 | 〈explicit sign〉 | @
     〈symbol element〉 →
         〈any character other than 〈vertical line〉or \〉
         | 〈inline hex escape〉 | 〈mnemonic escape〉 | \|


Marko

[toc] | [prev] | [next] | [standalone]

#110991

On 03/07/2016 12:01, Marko Rauhamaa wrote:
> Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid>:
>
>> It would be very confusing to have a variable named ∇f, as confusing
>> as naming a variable a+b or √x.
>
> Scheme allows *any* characters whatsoever in identifiers.

I think it's one of those languages that has already dispensed with most 
syntax anyway. Including distinctions between names and symbols.

Some people think that extra syntax rules including enforcing such 
distinctions and having restrictions can improve readability. Otherwise 
you can be looking at:

   a b c d e f g h

(not Scheme) and wondering which are names and which are operators.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#111013

On Sunday, July 3, 2016 at 11:50:52 PM UTC+12, BartC wrote:
> Otherwise you can be looking at:
> 
>    a b c d e f g h
> 
> (not Scheme) and wondering which are names and which are operators.

I did a language design for my MSc thesis where all “functions” were operators. So a construct like “f(a, b, c)” was really a monadic operator “f” followed by a single argument, a record constructor “(a, b, c)”.

[toc] | [prev] | [next] | [standalone]

#111021

On Sun, Jul 3, 2016 at 6:58 AM, John Ladasky <john_ladasky@sbcglobal.net> wrote:
> The nabla symbol (∇) is used in the naming of gradients. Python isn't having it.
> The interpreter throws a "SyntaxError: invalid character in identifier" when it
> encounters the ∇.

Del is a mathematical operator to take the gradient. It's not part of
the name. For `∇f`, the operator is `∇` and the function name is `f`.
Python lacks a mechanism to add user-defined operators. (R has this
capability.) Maybe this feature could be added. To make parsing
simple, user-defined operators could be limited to non-ASCII symbol
characters (math and other -- Sm, So). That simple option is off the
table if we allow symbol characters in names.

Adding an operator to the language itself requires a PEP. Recently PEP
465 added an `@` operator for matrix products. For example:

    >>> x = np.array([1j, 1])
    >>> x @ x
    0j
    >>> x @ x.conj() # Hermitian inner product
    (2+0j)

Note that using a non-ASCII operator was ruled out:

http://legacy.python.org/dev/peps/pep-0465/#choice-of-operator

[toc] | [prev] | [next] | [standalone]

#111024

On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
> Python lacks a mechanism to add user-defined operators. (R has this
> capability.) Maybe this feature could be added.

That would be neat. But remember, you would have to define the operator precedence as well. So you could no longer use a recursive-descent parser.

[toc] | [prev] | [next] | [standalone]

#111028

On 04/07/2016 01:00, Lawrence D’Oliveiro wrote:
> On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
>> Python lacks a mechanism to add user-defined operators. (R has this
>> capability.) Maybe this feature could be added.
>
> That would be neat. But remember, you would have to define the operator precedence as well. So you could no longer use a recursive-descent parser.

That wouldn't be a problem provided the new operator symbol and its 
precedence is known at a compile time, and defined before use.


-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#111032

On Monday, July 4, 2016 at 12:17:47 PM UTC+12, BartC wrote:
>
> On 04/07/2016 01:00, Lawrence D’Oliveiro wrote:
>>
>> On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
>>>
>>> Python lacks a mechanism to add user-defined operators. (R has this
>>> capability.) Maybe this feature could be added.
>>
>> That would be neat. But remember, you would have to define the operator
>> precedence as well. So you could no longer use a recursive-descent parser.
> 
> That wouldn't be a problem provided the new operator symbol and its 
> precedence is known at a compile time, and defined before use.

That is how it is normally done. (E.g. Algol 68.)

But you still couldn’t use a recursive-descent parser.

[toc] | [prev] | [next] | [standalone]

#111033

On 04/07/2016 01:24, Lawrence D’Oliveiro wrote:
> On Monday, July 4, 2016 at 12:17:47 PM UTC+12, BartC wrote:
>>
>> On 04/07/2016 01:00, Lawrence D’Oliveiro wrote:
>>>
>>> On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
>>>>
>>>> Python lacks a mechanism to add user-defined operators. (R has this
>>>> capability.) Maybe this feature could be added.
>>>
>>> That would be neat. But remember, you would have to define the operator
>>> precedence as well. So you could no longer use a recursive-descent parser.
>>
>> That wouldn't be a problem provided the new operator symbol and its
>> precedence is known at a compile time, and defined before use.
>
> That is how it is normally done. (E.g. Algol 68.)
>
> But you still couldn’t use a recursive-descent parser.

Why not?

The structure of such a parser doesn't need to exactly match the grammar 
with a dedicated block of code for each operator precedence. It can be 
table-driven so that an operator precedence value is just an attribute.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#111034

On Monday, July 4, 2016 at 12:40:14 PM UTC+12, BartC wrote:
> The structure of such a parser doesn't need to exactly match the grammar 
> with a dedicated block of code for each operator precedence. It can be 
> table-driven so that an operator precedence value is just an attribute.

Of course. But that’s not a recursive-descent parser any more.

[toc] | [prev] | [next] | [standalone]

#111037

On Sun, Jul 3, 2016, at 21:15, Lawrence D’Oliveiro wrote:
> On Monday, July 4, 2016 at 12:40:14 PM UTC+12, BartC wrote:
> > The structure of such a parser doesn't need to exactly match the grammar 
> > with a dedicated block of code for each operator precedence. It can be 
> > table-driven so that an operator precedence value is just an attribute.
> 
> Of course. But that’s not a recursive-descent parser any more.

It's still recursive descent if it, for example, calls the _same_ block
of code recursively with arguments to tell it which operator is being
considered. This would be analogous to, in Python, implementing a
recursive-descent parser with arbitrary callable objects instead of
simple functions.

[toc] | [prev] | [next] | [standalone]

#111055

On Monday, July 4, 2016 at 3:56:43 PM UTC+5:30, BartC wrote:
> On 04/07/2016 02:15, Lawrence D’Oliveiro wrote:
> > On Monday, July 4, 2016 at 12:40:14 PM UTC+12, BartC wrote:
> >> The structure of such a parser doesn't need to exactly match the grammar
> >> with a dedicated block of code for each operator precedence. It can be
> >> table-driven so that an operator precedence value is just an attribute.
> >
> > Of course. But that’s not a recursive-descent parser any more.
> >
> 
> All the parsers I write work the same way. If I can't describe them as 
> recursive descent, then I don't know what they are.
> 
> This is just recognising that a bunch of specialised functions that are 
> very similar can be reduced to one or two more generalised ones.

In gofer (likewise Haskell) one can concoct any operator and give it a precedence
and associativity -- l,r,non

Internals of Haskell I do not know, but of gofer I can say the following:

Implementation is in C.
Uses yacc to parse all operators left-assoc, same precedence
Then post-processes the tree with an elegant little shift-reduce parser
based on specified precedences and associativities.

I sometimes teach this to my kids as an example of how 
FP-style comments can clarify arcane imperative code:

Mark Jones (gofer author) original version + My version made executable
http://blog.languager.org/2016/07/a-little-functional-parser.html

[toc] | [prev] | [next] | [standalone]

#111066

On 04/07/2016 02:15, Lawrence D’Oliveiro wrote:
> On Monday, July 4, 2016 at 12:40:14 PM UTC+12, BartC wrote:
>> The structure of such a parser doesn't need to exactly match the grammar
>> with a dedicated block of code for each operator precedence. It can be
>> table-driven so that an operator precedence value is just an attribute.
>
> Of course. But that’s not a recursive-descent parser any more.
>

All the parsers I write work the same way. If I can't describe them as 
recursive descent, then I don't know what they are.

This is just recognising that a bunch of specialised functions that are 
very similar can be reduced to one or two more generalised ones.

-- 
bartc

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web