Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #93760 > unrolled thread

Re: Possibly Pythonic Tail Call Optimization (TCO/TRE)

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 00:05 +1000
    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Steven D'Aprano <steve@pearwood.info> - 2015-07-14 13:20 +1000
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-14 09:26 +0200
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ian Kelly <ian.g.kelly@gmail.com> - 2015-07-14 02:34 -0600
        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Grant Edwards <invalid@invalid.invalid> - 2015-07-14 14:03 +0000
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-14 12:43 +0200
    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Paul Rubin <no.email@nospam.invalid> - 2015-07-13 22:15 -0700
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 15:25 +1000
        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Paul Rubin <no.email@nospam.invalid> - 2015-07-13 22:41 -0700
          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 21:43 +1000
            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 15:28 +0300
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 22:55 +1000
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 16:38 +0300
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 23:43 +1000
                    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 20:29 +0300
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Steven D'Aprano <steve@pearwood.info> - 2015-07-15 03:48 +1000
                        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-15 10:28 +0200
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-14 16:02 +0200
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-15 00:07 +1000
                    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 20:31 +0300
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Terry Reedy <tjreedy@udel.edu> - 2015-07-14 20:41 -0400
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-15 11:10 +0200
                    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-16 10:43 +1200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-16 09:31 +0200
                        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-17 10:41 +1200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-16 21:45 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-16 15:56 +0200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 00:00 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-16 16:32 +0200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 02:43 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Joonas Liik <liik.joonas@gmail.com> - 2015-07-16 19:50 +0300
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 03:03 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ethan Furman <ethan@stoneleaf.us> - 2015-07-16 10:04 -0700
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Joonas Liik <liik.joonas@gmail.com> - 2015-07-16 20:34 +0300
                        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Steven D'Aprano <steve@pearwood.info> - 2015-07-17 04:58 +1000
                          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Joonas Liik <liik.joonas@gmail.com> - 2015-07-16 22:14 +0300
                            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-17 11:41 +1200
                          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ethan Furman <ethan@stoneleaf.us> - 2015-07-16 12:52 -0700
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 03:49 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Joonas Liik <liik.joonas@gmail.com> - 2015-07-16 21:23 +0300
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 04:29 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Terry Reedy <tjreedy@udel.edu> - 2015-07-16 16:19 -0400
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-17 12:48 +0200
                        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-19 10:57 +1200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 21:05 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-17 13:43 +0200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 21:49 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-17 14:54 +0200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-17 23:12 +1000
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Terry Reedy <tjreedy@udel.edu> - 2015-07-17 16:27 -0400
            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-15 21:13 +1200
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-15 12:55 +0300
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-07-15 12:22 +0100
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-16 10:34 +1200
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-07-16 00:34 +0100
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) MRAB <python@mrabarnett.plus.com> - 2015-07-15 13:00 +0100
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-15 22:13 +1000
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-16 10:34 +1200
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-15 22:22 +1000
        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 08:41 +0300
          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 21:33 +1000
            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 15:08 +0300
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-14 15:31 +1000
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ian Kelly <ian.g.kelly@gmail.com> - 2015-07-13 23:46 -0600
        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 08:57 +0300
          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-14 18:29 +1200
            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 10:05 +0300
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-07-15 17:52 +1200
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-15 09:44 +0300
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ned Batchelder <ned@nedbatchelder.com> - 2015-07-15 03:45 -0700
                    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-15 13:55 +0300
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2015-07-15 13:04 +0200
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ned Batchelder <ned@nedbatchelder.com> - 2015-07-15 04:12 -0700
                        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-15 14:24 +0300
                          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Rustom Mody <rustompmody@gmail.com> - 2015-07-16 21:28 -0700
            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-07-14 08:47 +0100
          Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ian Kelly <ian.g.kelly@gmail.com> - 2015-07-14 02:13 -0600
            Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 11:33 +0300
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ian Kelly <ian.g.kelly@gmail.com> - 2015-07-14 07:15 -0600
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 20:27 +0300
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Ian Kelly <ian.g.kelly@gmail.com> - 2015-07-15 08:28 -0600
                    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-15 17:55 +0300
              Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Steven D'Aprano <steve@pearwood.info> - 2015-07-15 03:34 +1000
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 20:43 +0300
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-15 03:52 +1000
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Steven D'Aprano <steve@pearwood.info> - 2015-07-15 04:06 +1000
                    Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 21:37 +0300
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-15 04:55 +1000
                        Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Marko Rauhamaa <marko@pacujo.net> - 2015-07-14 22:12 +0300
                      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Terry Reedy <tjreedy@udel.edu> - 2015-07-14 21:36 -0400
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Terry Reedy <tjreedy@udel.edu> - 2015-07-14 21:01 -0400
                  Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Chris Angelico <rosuav@gmail.com> - 2015-07-15 13:05 +1000
                Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Paul Rubin <no.email@nospam.invalid> - 2015-07-14 10:54 -0700
      Re: Possibly Pythonic Tail Call Optimization (TCO/TRE) Terry Reedy <tjreedy@udel.edu> - 2015-07-14 20:23 -0400

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

#93947

On Fri, Jul 17, 2015 at 4:23 AM, Joonas Liik <liik.joonas@gmail.com> wrote:
> On 16 July 2015 at 20:49, Chris Angelico <rosuav@gmail.com> wrote:
>>
>> This sounds like a denial-of-service attack. If you can state that no
>> reasonable document will ever have more than 100 levels of nesting,
>> then you can equally state that cutting the parser off with a tidy
>> exception if it exceeds 100 levels is safe.
>>
> This particular example does have that kind of smell.. my bad for
> being careless with examples.
>
> what if its not a ddos tho, maybe its just strange data?
>

That's why you're allowed to change the default limit either
direction. If you're guarding against a DOS, you can crank it down; if
you're working with something where 1000 stack entries isn't
unreasonable, you can crank it up. I honestly don't know what you'd
want to do if 5000+ stack entries isn't enough, but if you're working
with something *that* deeply nested, you probably know a lot more
about what you're doing than I ever will. Maybe you could recompile
CPython with a bigger stack? Give Jython or PyPy a try? No idea. But
I'm sure it'd be possible somehow.

ChrisA

[toc] | [prev] | [next] | [standalone]

#93963

On 7/16/2015 7:45 AM, Chris Angelico wrote:
> On Thu, Jul 16, 2015 at 5:31 PM, Antoon Pardon
> <antoon.pardon@rece.vub.ac.be> wrote:


>> Traceback are not the only or even the most useful
>> tool for debugging code. The current stack trace
>> doesn't even contain the value's of the variables
>> on the stack. So in case of Terry Reedy's example
>> that stack trace would IMO have been next to useless.
>
> Actually, they do contain all of that (at least, they do in Py3 - not
> sure about Py2 as I haven't checked). You can poke around with the
> locals at every point on the stack:
>
> def f(x):
>      if x < 10: g(x+10)
>      return 5
>
> def g(x):
>      if x % 3: h(x + 2)
>      return 7
>
> def h(x):
>      return 1/x
>
> try:
>      x = -12
>      print(f(x))
> except ZeroDivisionError as e:
>      tb = e.__traceback__
>      while tb:
>          fr = tb.tb_frame
>          print("In function %s (%s:%d), x = %r" % (
>              fr.f_code.co_name,
>              fr.f_code.co_filename,
>              fr.f_lineno,
>              fr.f_locals["x"],
>          ))
>          tb = tb.tb_next
>
>
> It's all there. And it's immensely helpful.

One of the features of Idle is Debug => Stack Viewer, which, when 
invoked immediately after an exception, displays a tree widget with a 
node for each stack frame in the traceback.  Running your code without 
the extra try: except: stuff, so a traceback is displayed, I easily see 
that x is -12, -2, and 0 in f, g, and h.  I suspect that this feature is 
not well known and is underused.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#94002

On 07/16/2015 06:43 PM, Chris Angelico wrote:
> On Fri, Jul 17, 2015 at 12:32 AM, Antoon Pardon
> <antoon.pardon@rece.vub.ac.be> wrote:
>
>> What is unclear about "as it is generally produced on stderr"? That you
>> can do a whole lot of stuff, doesn't mean that this whole lot of stuff is
>> part of what generally happens. When people on this list ask a person
>> to include the stacktrace with the description of the problem, they
>> don't mean something that includes the values of the variables.
> True. That said, though, it's not a justification for dropping stack
> frames; even in the form that's printed to stderr, there is immense
> value in them. It may be possible to explicitly drop frames that a
> programmer believes won't be useful, but a general and automated
> dropping of tail-call information will do more harm than good. The
> fact that some frameworks can show _even more_ helpful information out
> of a traceback only makes this stronger.

Just wondering, are traceback records of generators available? They are
if an exception is raised in the generator itself, but what if an exception
is raised in the loop that is driven by a generator. They don't appear in
the standard stack trace. 

It seems a bit strange that with the immense value that is given to 
stack frames, that these wouldn't be available somehow.

-- 
Antoon Pardon

[toc] | [prev] | [next] | [standalone]

#94067

Antoon Pardon wrote:
> It seems a bit strange that with the immense value that is given to 
> stack frames, that these wouldn't be available somehow.

There was discussion recently about the idea of providing
access to the frame of a suspended generator, so that
debuggers can inspect the stack of an asyncio task that's
not currently running.

The same mechanism ought to allow the state of an arbitrary
generator to be examined. You'd need to explicitly ask
for it, though -- it wouldn't appear in stack traces
automatically.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#94004

On Fri, Jul 17, 2015 at 8:48 PM, Antoon Pardon
<antoon.pardon@rece.vub.ac.be> wrote:
> Just wondering, are traceback records of generators available? They are
> if an exception is raised in the generator itself, but what if an exception
> is raised in the loop that is driven by a generator. They don't appear in
> the standard stack trace.

Not sure what you mean here. Something like this?

def gen():
    yield stuff
    yield more stuff

for stuff in gen():
    bomb with exception

The error didn't happen in the generator, so I wouldn't expect to see
it in the traceback. There's still room for the cause of an error to
not be in the traceback; imagine, for instance, a function that
populates a concrete list, and then you iterate over the list. If that
function sticks a None into the list and the subsequent processing is
expecting all strings, that's going to bomb, but then you have to
figure out where the None came from. If the traceback could include
that, it'd be great, but some things aren't possible. Doesn't mean
we're happy to sacrifice other functionality.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94006

On 07/17/2015 01:05 PM, Chris Angelico wrote:
> On Fri, Jul 17, 2015 at 8:48 PM, Antoon Pardon
> <antoon.pardon@rece.vub.ac.be> wrote:
>> Just wondering, are traceback records of generators available? They are
>> if an exception is raised in the generator itself, but what if an exception
>> is raised in the loop that is driven by a generator. They don't appear in
>> the standard stack trace.
> Not sure what you mean here. Something like this?
>
> def gen():
>     yield stuff
>     yield more stuff
>
> for stuff in gen():
>     bomb with exception
>
> The error didn't happen in the generator, so I wouldn't expect to see
> it in the traceback.

Yes something like that. And I wouldn't expect it either but if it
is not present, is it because nobody thought about it or because it
is a bad idea or an idea difficult to implement?

> There's still room for the cause of an error to
> not be in the traceback; imagine, for instance, a function that
> populates a concrete list, and then you iterate over the list. If that
> function sticks a None into the list and the subsequent processing is
> expecting all strings, that's going to bomb, but then you have to
> figure out where the None came from. If the traceback could include
> that, it'd be great, but some things aren't possible.

Sure, but in this case, the generator is still active. The Runtime
would be able to jump to and somehow activates it's stack record
for the next value. So why would we expect it to be impossible to
include this trace back record in a stack trace?

>  Doesn't mean
> we're happy to sacrifice other functionality.

Indeed, this is an independend problem. Whatever the answer here doesn't
need to affect how one feels about losing trace back record because of
TCO.

[toc] | [prev] | [next] | [standalone]

#94007

On Fri, Jul 17, 2015 at 9:43 PM, Antoon Pardon
<antoon.pardon@rece.vub.ac.be> wrote:
> On 07/17/2015 01:05 PM, Chris Angelico wrote:
>> On Fri, Jul 17, 2015 at 8:48 PM, Antoon Pardon
>> <antoon.pardon@rece.vub.ac.be> wrote:
>>> Just wondering, are traceback records of generators available? They are
>>> if an exception is raised in the generator itself, but what if an exception
>>> is raised in the loop that is driven by a generator. They don't appear in
>>> the standard stack trace.
>> Not sure what you mean here. Something like this?
>>
>> def gen():
>>     yield stuff
>>     yield more stuff
>>
>> for stuff in gen():
>>     bomb with exception
>>
>> The error didn't happen in the generator, so I wouldn't expect to see
>> it in the traceback.
>
> Yes something like that. And I wouldn't expect it either but if it
> is not present, is it because nobody thought about it or because it
> is a bad idea or an idea difficult to implement?
>
>> There's still room for the cause of an error to
>> not be in the traceback; imagine, for instance, a function that
>> populates a concrete list, and then you iterate over the list. If that
>> function sticks a None into the list and the subsequent processing is
>> expecting all strings, that's going to bomb, but then you have to
>> figure out where the None came from. If the traceback could include
>> that, it'd be great, but some things aren't possible.
>
> Sure, but in this case, the generator is still active. The Runtime
> would be able to jump to and somehow activates it's stack record
> for the next value. So why would we expect it to be impossible to
> include this trace back record in a stack trace?

Python could also give you stack traces for any other threads that are
concurrently running, on the off-chance that one of them affected it.
But the only influence the generator has on the loop is to yield a
value or signal termination; if an exception is thrown in the loop
itself, the local name 'stuff' should have all the information about
that cause. Python isn't a mind-reader, no matter how much it may look
like one, and it can't know that this function's return value should
be shown as part of a completely different function's stack trace.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94010

On 07/17/2015 01:49 PM, Chris Angelico wrote:
> On Fri, Jul 17, 2015 at 9:43 PM, Antoon Pardon
> <antoon.pardon@rece.vub.ac.be> wrote:
>
>
>> Sure, but in this case, the generator is still active. The Runtime
>> would be able to jump to and somehow activates it's stack record
>> for the next value. So why would we expect it to be impossible to
>> include this trace back record in a stack trace?
> Python could also give you stack traces for any other threads that are
> concurrently running, on the off-chance that one of them affected it.
> But the only influence the generator has on the loop is to yield a
> value or signal termination; if an exception is thrown in the loop
> itself, the local name 'stuff' should have all the information about
> that cause.

But the local name 'stuff' may only have the information for the immediate
cause. The underlying cause may be available in the generator. Suppose
you have a generator that should only generate positive numbers that you
use to divide some other number by. Your loop crashes because of a DivideByZeroError
Sure the local name shows the dividor to be zero, but you have no
information on why your generator produced a zero, but there may be a
clue in the trace back record of the generator.

>  Python isn't a mind-reader, no matter how much it may look
> like one, and it can't know that this function's return value should
> be shown as part of a completely different function's stack trace.

It is not a matter of mindreading. And it is not a completely different
functions stack trace. It is the trace back record of a generator that
is used by the process/thread that crashed. And AFAIK an active generator
belongs to one specific thread. You can't have it yield a value to a different
thread and you can't send it a value from an other thread. So I really
see no reason to exclude the trace back records of active generators
from a stack trace of a crashed thread.

-- 
Antoon Pardon

[toc] | [prev] | [next] | [standalone]

#94011

On Fri, Jul 17, 2015 at 10:54 PM, Antoon Pardon
<antoon.pardon@rece.vub.ac.be> wrote:
> On 07/17/2015 01:49 PM, Chris Angelico wrote:
>> On Fri, Jul 17, 2015 at 9:43 PM, Antoon Pardon
>> <antoon.pardon@rece.vub.ac.be> wrote:
>>
>>
>>> Sure, but in this case, the generator is still active. The Runtime
>>> would be able to jump to and somehow activates it's stack record
>>> for the next value. So why would we expect it to be impossible to
>>> include this trace back record in a stack trace?
>> Python could also give you stack traces for any other threads that are
>> concurrently running, on the off-chance that one of them affected it.
>> But the only influence the generator has on the loop is to yield a
>> value or signal termination; if an exception is thrown in the loop
>> itself, the local name 'stuff' should have all the information about
>> that cause.
>
> But the local name 'stuff' may only have the information for the immediate
> cause. The underlying cause may be available in the generator. Suppose
> you have a generator that should only generate positive numbers that you
> use to divide some other number by. Your loop crashes because of a DivideByZeroError
> Sure the local name shows the dividor to be zero, but you have no
> information on why your generator produced a zero, but there may be a
> clue in the trace back record of the generator.

Indeed, but there's nothing special about generators here. The same
sequence could have been a concrete list, or it could have been some
other kind of iterator (any object with __iter__ and __next__), which
won't have a stack frame. Special cases aren't special enough to warp
exception handling around.

>>  Python isn't a mind-reader, no matter how much it may look
>> like one, and it can't know that this function's return value should
>> be shown as part of a completely different function's stack trace.
>
> It is not a matter of mindreading. And it is not a completely different
> functions stack trace. It is the trace back record of a generator that
> is used by the process/thread that crashed. And AFAIK an active generator
> belongs to one specific thread. You can't have it yield a value to a different
> thread and you can't send it a value from an other thread. So I really
> see no reason to exclude the trace back records of active generators
> from a stack trace of a crashed thread.

No, generators are fine across threads:

rosuav@sikorsky:~$ python3 threadgen.py
Starting!
First yielded value
Continuing!
Second yielded value
Terminating.
Traceback (most recent call last):
  File "threadgen.py", line 20, in <module>
    print(next(gen))
StopIteration
rosuav@sikorsky:~$ cat threadgen.py
import threading
import time

def thread():
    time.sleep(0.5)
    print(next(gen))

threading.Thread(target=thread).start()

def generator():rosuav@sikorsky:~$

    print("Starting!")
    yield "First yielded value"
    print("Continuing!")
    yield "Second yielded value"
    print("Terminating.")

gen = generator()
print(next(gen))
time.sleep(1)
print(next(gen))
rosuav@sikorsky:~$

In fact, a generator doesn't have a stack unless it's currently
executing, so all you could get is whatever's actually inside it (that
is, if there's a deep tree of 'yield from's, you could dig up that
part of the stack). I'm not sure this would really help you very much.

ChrisA

[toc] | [prev] | [next] | [standalone]

#94030

On 7/17/2015 7:43 AM, Antoon Pardon wrote:
> On 07/17/2015 01:05 PM, Chris Angelico wrote:

>> def gen():
>>      yield stuff
>>      yield more stuff
>>
>> for stuff in gen():
>>      bomb with exception
>>
>> The error didn't happen in the generator, so I wouldn't expect to see
>> it in the traceback.
>
> Yes something like that. And I wouldn't expect it either but if it
> is not present, is it because nobody thought about it or because it
> is a bad idea or an idea difficult to implement?
>
>> There's still room for the cause of an error to
>> not be in the traceback; imagine, for instance, a function that
>> populates a concrete list, and then you iterate over the list. If that
>> function sticks a None into the list and the subsequent processing is
>> expecting all strings, that's going to bomb, but then you have to
>> figure out where the None came from. If the traceback could include
>> that, it'd be great, but some things aren't possible.
>
> Sure, but in this case, the generator is still active.

No more than any other object sitting around inactive. Calling a 
generator function like gen above returns a generator with the generator 
function, in a suspended inactive state, attached as an attribute.  When 
the generator.__next__ function is called, it activates its instance of 
the generator function, which suspends itself again after yielding 
something.  At the point of the exception above, the generator next 
function has returned.  There could be multiple generators with 
suspended generator functions sitting around.  For instance:

def f():
     for tup in zip(gf0, gf1, gf2, gf3, gf4, gf5):
         a = tup[6]  # whoops, IndexError
         <unreachable code meant to do something>

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#93856

Chris Angelico wrote:
> I'm still interested in the explicit "replace current stack frame with
> this call" operation. Calling it "goto" seems wrong, as most languages
> with goto restrict it to _within_ a function,

This just suggests to me is that most language designers
are not very imaginative. :-)

A tail call *is* a goto. That's how you implement one in
assembly language -- you write a jump instruction instead
of a call instruction. The jump doesn't have to be to
the same function.

Also see:

https://litrev.wordpress.com/2009/07/16/lambda-the-ultimate-goto/

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#93859

Gregory Ewing <greg.ewing@canterbury.ac.nz>:

> A tail call *is* a goto. That's how you implement one in assembly
> language -- you write a jump instruction instead of a call
> instruction. The jump doesn't have to be to the same function.

In functional programming languages you might not even have a call
stack. Instead, you specify the program execution through
transformations. When you evaluate a program, you repeatedly convert the
program text into new programs until no more transformations apply and
the execution halts with the result of the computation.


Marko

[toc] | [prev] | [next] | [standalone]

#93866

On 15/07/2015 10:13, Gregory Ewing wrote:
> Chris Angelico wrote:
>> I'm still interested in the explicit "replace current stack frame with
>> this call" operation. Calling it "goto" seems wrong, as most languages
>> with goto restrict it to _within_ a function,
>
> This just suggests to me is that most language designers
> are not very imaginative. :-)
>
> A tail call *is* a goto. That's how you implement one in
> assembly language -- you write a jump instruction instead
> of a call instruction. The jump doesn't have to be to
> the same function.
>

IIRC the realms of the C setjmp and longjmp.  I'm just wondering out 
aloud if anybody has ever tried playing with cPython using these, or 
whether the code and/or Python itself is just too complex to allow this 
to even be tried, let alone succeed.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#93884

Mark Lawrence wrote:
> IIRC the realms of the C setjmp and longjmp.

Not really the same thing. A longjmp chops the stack
back, whereas a tail call avoids putting something on
the stack to begin with.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#93887

On 15/07/2015 23:34, Gregory Ewing wrote:
> Mark Lawrence wrote:
>> IIRC the realms of the C setjmp and longjmp.
>
> Not really the same thing. A longjmp chops the stack
> back, whereas a tail call avoids putting something on
> the stack to begin with.
>

Thanks for that :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#93869

On 2015-07-15 12:22, Mark Lawrence wrote:
> On 15/07/2015 10:13, Gregory Ewing wrote:
>> Chris Angelico wrote:
>>> I'm still interested in the explicit "replace current stack frame with
>>> this call" operation. Calling it "goto" seems wrong, as most languages
>>> with goto restrict it to _within_ a function,
>>
>> This just suggests to me is that most language designers
>> are not very imaginative. :-)
>>
>> A tail call *is* a goto. That's how you implement one in
>> assembly language -- you write a jump instruction instead
>> of a call instruction. The jump doesn't have to be to
>> the same function.
>>
>
> IIRC the realms of the C setjmp and longjmp.  I'm just wondering out
> aloud if anybody has ever tried playing with cPython using these, or
> whether the code and/or Python itself is just too complex to allow this
> to even be tried, let alone succeed.
>
The problem with longjmp is that it could inadvertently bypass some
clean-up code or deallocation, or, in the case of CPython, a DECREF.

[toc] | [prev] | [next] | [standalone]

#93871

On Wed, Jul 15, 2015 at 10:00 PM, MRAB <python@mrabarnett.plus.com> wrote:
> On 2015-07-15 12:22, Mark Lawrence wrote:
>>
>> On 15/07/2015 10:13, Gregory Ewing wrote:
>>>
>>> Chris Angelico wrote:
>>>>
>>>> I'm still interested in the explicit "replace current stack frame with
>>>> this call" operation. Calling it "goto" seems wrong, as most languages
>>>> with goto restrict it to _within_ a function,
>>>
>>>
>>> This just suggests to me is that most language designers
>>> are not very imaginative. :-)
>>>
>>> A tail call *is* a goto. That's how you implement one in
>>> assembly language -- you write a jump instruction instead
>>> of a call instruction. The jump doesn't have to be to
>>> the same function.
>>>
>>
>> IIRC the realms of the C setjmp and longjmp.  I'm just wondering out
>> aloud if anybody has ever tried playing with cPython using these, or
>> whether the code and/or Python itself is just too complex to allow this
>> to even be tried, let alone succeed.
>>
> The problem with longjmp is that it could inadvertently bypass some
> clean-up code or deallocation, or, in the case of CPython, a DECREF.

Which really says that TCO is impossible if you have any sort of
clean-up or deallocation to be done after the call begins. The only
way to truly turn your tail call into a GOTO is to do all your cleanup
first. "try: return x() finally: more_code" is not a tail call, nor is
it if you have "except:" in place of "finally:".

ChrisA

[toc] | [prev] | [next] | [standalone]

#93885

Chris Angelico wrote:
> Which really says that TCO is impossible if you have any sort of
> clean-up or deallocation to be done after the call begins. The only
> way to truly turn your tail call into a GOTO is to do all your cleanup
> first.

Indeed. In compilers that implement TCO, there's quite
a lot more to it than just "replace CALL with JMP". It
requires rethinking your whole strategy on when to put
various things on the stack and take them off again,
so that by the time you get to the point of the tail
call, there is nothing on the stack other than the
arguments to the tail call and the ultimate return
address.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#93872

On Wed, Jul 15, 2015 at 7:13 PM, Gregory Ewing
<greg.ewing@canterbury.ac.nz> wrote:
> Chris Angelico wrote:
>>
>> I'm still interested in the explicit "replace current stack frame with
>> this call" operation. Calling it "goto" seems wrong, as most languages
>> with goto restrict it to _within_ a function,
>
>
> This just suggests to me is that most language designers
> are not very imaginative. :-)
>
> A tail call *is* a goto. That's how you implement one in
> assembly language -- you write a jump instruction instead
> of a call instruction. The jump doesn't have to be to
> the same function.
>

Sure it is; but then, a while loop is a goto, but we don't call them
goto loops. Ultimately, assembly language lets you treat everything
the same way - in some CPUs, a jump is an assignment to the
instruction pointer register, and is thus the same kind of operation
as any other. In real-mode Intel 80x86 Assembly, there are
unconditional and conditional jumps, where conditional jumps are
limited to the shortest possible form of jump (relative jumps within
+/- 128 bytes), but other than that, there's no distinctions between
"if" statements, "while" loops, etc, etc. (Yes, there's also
near-call, far-call, and interrupt, which push the instruction
pointer, the IP plus the code segment, and the IP, CS, and the flags
register, respectively - but then still just do a straight jump to the
other location.) But in high level languages, we differentiate all of
them, because the human who reads the code should differentiate
between them. You don't use Python's "while" statement as a sort of
"if", nor vice versa. At least, not usually...

while x < 1:
    print("x is less than one!")
    break
else:
    print("x is one or greater!")

ChrisA

[toc] | [prev] | [next] | [standalone]

#93785

Chris Angelico <rosuav@gmail.com>:

>>   def quicksort(array, start, end):
>>      midp = partition(array, start, end)
>>      if midp <= (start+end)//2:
>>         quicksort(array, start, midp)
>>         quicksort(array, midp+1, end)
>>      else:
>>         quicksort(array, midp+1, end)
>>         quicksort(array, start, midp)
> [...]
> That's a prime example of recursion... but not of recursion that can
> be tail-call optimized into iteration.

Of course it can. The 2nd and 4th calls to quicksort can be trivially
tail-call-optimized.

Tail-call optimization has nothing to do with converting algorithms into
iterations. It's a prosaic trick of dropping an unneeded stack frame
before making a function call.

> The claim that TCO means you don't need stack space for all those
> levels of recursion doesn't work if you still need stack space for all
> those levels of recursion *before* you get to the tail call.

Nobody is making that claim.

What you are questioning is what tail-call optimization would buy you in
practice. Not much. The few times you run into a stack overflow, you can
refactor your code pretty easily. However, when you have to do that, it
feels a bit silly.


Marko

[toc] | [prev] | [next] | [standalone]

Page 3 of 5 — ← Prev page 1 2 [3] 4 5  Next page →

Back to top | Article view | comp.lang.python

csiph-web