Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #110978 > unrolled thread

Well, I finally ran into a Python Unicode problem, sort of

Back to article view | Back to comp.lang.python

Contents

  Well, I finally ran into a Python Unicode problem, sort of John Ladasky <john_ladasky@sbcglobal.net> - 2016-07-02 23:58 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 00:07 -0700
      Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 10:26 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 01:38 -0700
          Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 12:01 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of Chris Angelico <rosuav@gmail.com> - 2016-07-04 01:14 +1000
            Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 14:21 -0700
      Re: Well, I finally ran into a Python Unicode problem, sort of John Ladasky <john_ladasky@sbcglobal.net> - 2016-07-03 09:24 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of Rustom Mody <rustompmody@gmail.com> - 2016-07-03 00:26 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-03 10:29 +0300
      Re: Well, I finally ran into a Python Unicode problem, sort of Robert Kern <robert.kern@gmail.com> - 2016-07-03 10:16 +0100
    Re: Well, I finally ran into a Python Unicode problem, sort of Chris Angelico <rosuav@gmail.com> - 2016-07-03 17:41 +1000
      Re: Well, I finally ran into a Python Unicode problem, sort of John Ladasky <john_ladasky@sbcglobal.net> - 2016-07-03 11:39 -0700
        Re: Well, I finally ran into a Python Unicode problem, sort of MRAB <python@mrabarnett.plus.com> - 2016-07-03 20:41 +0100
        Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 14:28 -0700
          Re: Well, I finally ran into a Python Unicode problem, sort of Steven D'Aprano <steve@pearwood.info> - 2016-07-04 12:33 +1000
            Re: Well, I finally ran into a Python Unicode problem, sort of Rustom Mody <rustompmody@gmail.com> - 2016-07-03 22:14 -0700
              Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-04 01:11 -0700
                Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-04 13:10 +0300
                Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-04 12:04 +0300
              Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-04 09:08 +0300
    Re: Well, I finally ran into a Python Unicode problem, sort of Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2016-07-03 11:53 +0200
      Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 14:01 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of Christian Gollwitzer <auriocus@gmx.de> - 2016-07-03 13:16 +0200
          Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 14:22 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of Christian Gollwitzer <auriocus@gmx.de> - 2016-07-03 16:05 +0200
              Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-03 17:39 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of Random832 <random832@fastmail.com> - 2016-07-03 16:49 -0400
              Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-04 00:25 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-03 12:50 +0100
          Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 14:26 -0700
    Re: Well, I finally ran into a Python Unicode problem, sort of eryk sun <eryksun@gmail.com> - 2016-07-03 23:46 +0000
      Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 17:00 -0700
        Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 01:17 +0100
          Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 17:24 -0700
            Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 01:39 +0100
              Re: Well, I finally ran into a Python Unicode problem, sort of Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-07-03 18:15 -0700
                Re: Well, I finally ran into a Python Unicode problem, sort of Random832 <random832@fastmail.com> - 2016-07-03 22:20 -0400
                Re: Well, I finally ran into a Python Unicode problem, sort of Rustom Mody <rustompmody@gmail.com> - 2016-07-04 05:16 -0700
                Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 11:26 +0100
          Re: Well, I finally ran into a Python Unicode problem, sort of Steven D'Aprano <steve@pearwood.info> - 2016-07-04 12:30 +1000
            Re: Well, I finally ran into a Python Unicode problem, sort of Ned Batchelder <ned@nedbatchelder.com> - 2016-07-04 05:47 -0700
              Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 15:36 +0100
                Re: Well, I finally ran into a Python Unicode problem, sort of Ned Batchelder <ned@nedbatchelder.com> - 2016-07-04 07:46 -0700
                  Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 17:16 +0100
                    Re: Well, I finally ran into a Python Unicode problem, sort of Marko Rauhamaa <marko@pacujo.net> - 2016-07-04 19:44 +0300
            Re: Well, I finally ran into a Python Unicode problem, sort of BartC <bc@freeuk.com> - 2016-07-04 11:05 +0100
              Re: Well, I finally ran into a Python Unicode problem, sort of Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-07-04 13:26 +0300
        Re: Well, I finally ran into a Python Unicode problem, sort of Random832 <random832@fastmail.com> - 2016-07-03 22:16 -0400

Page 3 of 3 — ← Prev page 1 2 [3]

#111038

On Mon, 4 Jul 2016 10:17 am, BartC wrote:

> On 04/07/2016 01:00, Lawrence D’Oliveiro wrote:
>> On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
>>> Python lacks a mechanism to add user-defined operators. (R has this
>>> capability.) Maybe this feature could be added.
>>
>> That would be neat. But remember, you would have to define the operator
>> precedence as well. So you could no longer use a recursive-descent
>> parser.
> 
> That wouldn't be a problem provided the new operator symbol and its
> precedence is known at a compile time, and defined before use.

You're still having problems with the whole Python-as-a-dynamic-language
thing, aren't you? :-)

In full generality, you would want to be able to define unary prefix, unary
suffix and binary infix operators, and set their precedence and whether
they associate to the left or the right. That's probably a bit much to
expect.

But if we limit ourselves to the boring case of binary infix operators of a
single precedence and associtivity, there's a simple approach: the parser
can allow any unicode code point of category "Sm" as a legal operator, e.g.
x ∇ y. Pre-defined operators like + - * etc continue to call the same
dunder methods they already do, but anything else tries calling:

x.__oper__('∇', y)
y.__roper__('∇', x)

and if neither of those exist and return a result other than NotImplemented,
then finally raise a runtime TypeError('undefined operator ∇').

But I don't think this will ever be part of Python.



-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[toc] | [prev] | [next] | [standalone]

#111048

On Monday, July 4, 2016 at 6:05:20 AM UTC-4, BartC wrote:
> On 04/07/2016 03:30, Steven D'Aprano wrote:
> > On Mon, 4 Jul 2016 10:17 am, BartC wrote:
> >
> >> On 04/07/2016 01:00, Lawrence D’Oliveiro wrote:
> >>> On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
> >>>> Python lacks a mechanism to add user-defined operators. (R has this
> >>>> capability.) Maybe this feature could be added.
> >>>
> >>> That would be neat. But remember, you would have to define the operator
> >>> precedence as well. So you could no longer use a recursive-descent
> >>> parser.
> >>
> >> That wouldn't be a problem provided the new operator symbol and its
> >> precedence is known at a compile time, and defined before use.
> >
> > You're still having problems with the whole Python-as-a-dynamic-language
> > thing, aren't you? :-)
> 
> Well it isn't completely dynamic, not unless code only exists as a eval 
> or exec argument string (and even there, any changes will only be seen 
> on calling eval or exec again on the same string).
> 
> Most Pythons seem to pre-compile code before executing the result. That 
> pre-compilation requires that operators and precedences are known in 
> advance and the resulting instructions are then hard-coded before execution.

This is the key but subtle point that all the discussion of parser mechanics
are missing: Python today needs no information from imported modules in
order to compile a file.  When the compiler encounters "import xyzzy" in
a file, it doesn't have to do anything to find or read xyzzy.py at compile
time.

If operators can be invented, they will only be useful if they can be
created in modules which you then import and use.  But that would mean that
imported files would have to be found and read during compilation, not
during execution as they are now.

This is a huge change.

--Ned.

[toc] | [prev] | [next] | [standalone]

#111051

On 04/07/2016 13:47, Ned Batchelder wrote:
> On Monday, July 4, 2016 at 6:05:20 AM UTC-4, BartC wrote:
>> On 04/07/2016 03:30, Steven D'Aprano wrote:

>>> You're still having problems with the whole Python-as-a-dynamic-language
>>> thing, aren't you? :-)

>> Most Pythons seem to pre-compile code before executing the result. That
>> pre-compilation requires that operators and precedences are known in
>> advance and the resulting instructions are then hard-coded before execution.
>
> This is the key but subtle point that all the discussion of parser mechanics
> are missing: Python today needs no information from imported modules in
> order to compile a file.  When the compiler encounters "import xyzzy" in
> a file, it doesn't have to do anything to find or read xyzzy.py at compile
> time.

Yeah, there's that small detail. Anything affecting how source is to be 
parsed needs to known in advance.

> If operators can be invented, they will only be useful if they can be
> created in modules which you then import and use.  But that would mean that
> imported files would have to be found and read during compilation, not
> during execution as they are now.
>
> This is a huge change.

I've used a kind of 'weak' import scheme elsewhere, corresponding to C's 
'#include'.

Then the textual contents of that 'imported' module are read by the 
compiler, and treated as though they occurred in this module. No new 
namespace is created.

I think that could work in Python provided whatever is defined can 
tolerate having copies redefined in each module that includes the same 
file. Anything that is defined once and is never assigned to nor 
modified for example.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#111053

On Monday, July 4, 2016 at 10:36:54 AM UTC-4, BartC wrote:
> On 04/07/2016 13:47, Ned Batchelder wrote:
> > On Monday, July 4, 2016 at 6:05:20 AM UTC-4, BartC wrote:
> >> On 04/07/2016 03:30, Steven D'Aprano wrote:
> 
> >>> You're still having problems with the whole Python-as-a-dynamic-language
> >>> thing, aren't you? :-)
> 
> >> Most Pythons seem to pre-compile code before executing the result. That
> >> pre-compilation requires that operators and precedences are known in
> >> advance and the resulting instructions are then hard-coded before execution.
> >
> > This is the key but subtle point that all the discussion of parser mechanics
> > are missing: Python today needs no information from imported modules in
> > order to compile a file.  When the compiler encounters "import xyzzy" in
> > a file, it doesn't have to do anything to find or read xyzzy.py at compile
> > time.
> 
> Yeah, there's that small detail. Anything affecting how source is to be 
> parsed needs to known in advance.
> 
> > If operators can be invented, they will only be useful if they can be
> > created in modules which you then import and use.  But that would mean that
> > imported files would have to be found and read during compilation, not
> > during execution as they are now.
> >
> > This is a huge change.
> 
> I've used a kind of 'weak' import scheme elsewhere, corresponding to C's 
> '#include'.
> 
> Then the textual contents of that 'imported' module are read by the 
> compiler, and treated as though they occurred in this module. No new 
> namespace is created.
> 
> I think that could work in Python provided whatever is defined can 
> tolerate having copies redefined in each module that includes the same 
> file. Anything that is defined once and is never assigned to nor 
> modified for example.

You are hand-waving over huge details of semantics that are very important
in Python.  For example, it is very important not to have copies of
classes.  Importing a module must produce the same module object
everywhere it is imported, and the classes defined in the module must
be defined only once.

This is what makes catching exceptions work (because it is based on an
exception being an instance of a particular class), and what makes
class attributes shared among all the instances of the class.

--Ned.

[toc] | [prev] | [next] | [standalone]

#111077

On 04/07/2016 15:46, Ned Batchelder wrote:
> On Monday, July 4, 2016 at 10:36:54 AM UTC-4, BartC wrote:
>> On 04/07/2016 13:47, Ned Batchelder wrote:

>>> This is a huge change.
>>
>> I've used a kind of 'weak' import scheme elsewhere, corresponding to C's
>> '#include'.

>> I think that could work in Python provided whatever is defined can
>> tolerate having copies redefined in each module that includes the same
>> file. Anything that is defined once and is never assigned to nor
>> modified for example.
>
> You are hand-waving over huge details of semantics that are very important
> in Python.  For example, it is very important not to have copies of
> classes.  Importing a module must produce the same module object
> everywhere it is imported, and the classes defined in the module must
> be defined only once.

So that would be something that doesn't tolerate copies.

But I think that a bigger change for Python wouldn't be new ways of 
doing imports, but the concept of having a user-defined anything that is 
a constant at compile-time. And not part of a conditional statement either.

Usually anything that is defined can be changed at run-time so that the 
compiler can never assume anything.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#111079

BartC <bc@freeuk.com>:
> Usually anything that is defined can be changed at run-time so that the
> compiler can never assume anything.

The compiler can't assume anything permanent, but it could heuristically
make excellent guesses at runtime. It needs to verify its guesses at the
boundaries of compiled code and gradually keep expanding the boundaries.
If the guesses end up being wrong, it has to correct its assumptions and
recompile the relevant parts of the code.


Marko

[toc] | [prev] | [next] | [standalone]

#111056

On 04/07/2016 03:30, Steven D'Aprano wrote:
> On Mon, 4 Jul 2016 10:17 am, BartC wrote:
>
>> On 04/07/2016 01:00, Lawrence D’Oliveiro wrote:
>>> On Monday, July 4, 2016 at 11:47:26 AM UTC+12, eryk sun wrote:
>>>> Python lacks a mechanism to add user-defined operators. (R has this
>>>> capability.) Maybe this feature could be added.
>>>
>>> That would be neat. But remember, you would have to define the operator
>>> precedence as well. So you could no longer use a recursive-descent
>>> parser.
>>
>> That wouldn't be a problem provided the new operator symbol and its
>> precedence is known at a compile time, and defined before use.
>
> You're still having problems with the whole Python-as-a-dynamic-language
> thing, aren't you? :-)

Well it isn't completely dynamic, not unless code only exists as a eval 
or exec argument string (and even there, any changes will only be seen 
on calling eval or exec again on the same string).

Most Pythons seem to pre-compile code before executing the result. That 
pre-compilation requires that operators and precedences are known in 
advance and the resulting instructions are then hard-coded before execution.

> In full generality, you would want to be able to define unary prefix, unary
> suffix and binary infix operators, and set their precedence and whether
> they associate to the left or the right. That's probably a bit much to
> expect.

No, that's all possible. Maybe that's even how some language 
implementations work, defining all the set of standard operators at the 
start.

> But if we limit ourselves to the boring case of binary infix operators of a
> single precedence and associtivity, there's a simple approach: the parser
> can allow any unicode code point of category "Sm" as a legal operator, e.g.
> x ∇ y. Pre-defined operators like + - * etc continue to call the same
> dunder methods they already do, but anything else tries calling:
>
> x.__oper__('∇', y)
> y.__roper__('∇', x)
>
> and if neither of those exist and return a result other than NotImplemented,
> then finally raise a runtime TypeError('undefined operator ∇').

A simpler approach is to treat user-defined operators as aliases for 
functions:

def myadd(a,b):
	return a+b

operator ∇:
    (myadd,2,+3)   # map to myadd, 2 operands, prio 3, LTR

x = y ∇ z

is then equivalent to:

x = myadd(y,z)

However you will usually want to be able overload the same operator for 
different operand types. That means mapping the operator to one of 
several methods. Maybe even allowing the operator to have either one or 
two operands.

Trickier but still doable I think.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#111067

BartC writes:

> A simpler approach is to treat user-defined operators as aliases for
> functions:
>
> def myadd(a,b):
> 	return a+b
>
> operator ∇:
>    (myadd,2,+3)   # map to myadd, 2 operands, prio 3, LTR
>
> x = y ∇ z
>
> is then equivalent to:
>
> x = myadd(y,z)
>
> However you will usually want to be able overload the same operator
> for different operand types. That means mapping the operator to one of
> several methods. Maybe even allowing the operator to have either one
> or two operands.
>
> Trickier but still doable I think.

Julia does something like that. The parser knows a number of symbols
that it treats as operators, some of them are aliases for ASCII names,
all operators correspond to generic functions, and the programmer can
add methods for their own types (or for pre-existing types) to these
functions.

Prolog opens its precedence table for the programmer. I don't know if
there's been any Unicode activity, or any activity, in recent years, but
there are actually two different issues here: what is parsed as an
identifier, and what identifiers are treated as operator symbols (with
what precedence and associativity).

[toc] | [prev] | [next] | [standalone]

#111036

On Sun, Jul 3, 2016, at 20:00, Lawrence D’Oliveiro wrote:
> That would be neat. But remember, you would have to define the operator
> precedence as well. So you could no longer use a recursive-descent
> parser.

You could use a recursive-descent parser if you monkey-patch the parser
when adding a new operator.

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.python

csiph-web