Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109055 > unrolled thread

Spurious issue in CPython 2.7.5

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Spurious issue in CPython 2.7.5 "thomas povtal.org" <thomas@povtal.org> - 2016-05-24 12:22 +0200
    Re: Spurious issue in CPython 2.7.5 Novocastrian_Nomad <gregory.j.baker@gmail.com> - 2016-05-24 08:05 -0700
      Re: Spurious issue in CPython 2.7.5 "thomas povtal.org" <thomas@povtal.org> - 2016-05-24 17:20 +0200
    Re: Spurious issue in CPython 2.7.5 Steven D'Aprano <steve@pearwood.info> - 2016-05-25 03:12 +1000
      Re: Spurious issue in CPython 2.7.5 Chris Angelico <rosuav@gmail.com> - 2016-05-25 03:23 +1000
      Re: Spurious issue in CPython 2.7.5 "thomas povtal.org" <thomas@povtal.org> - 2016-05-25 14:04 +0200
      Re: Spurious issue in CPython 2.7.5 Tim Golden <mail@timgolden.me.uk> - 2016-05-25 13:13 +0100
      Re: Spurious issue in CPython 2.7.5 "thomas povtal.org" <thomas@povtal.org> - 2016-05-25 14:26 +0200

#109055 — Spurious issue in CPython 2.7.5

   Hi,

   Please excuse me if this is not the right place, but I have some issues
   with CPython on a NUMA machine.

   1: I get "RuntimeWarning: tp_compare didn't return -1 or -2 for
   exception". It's a line like:

   "if Foo = False:" where Foo is a global variable (global Foo).

   Now, I've searched somewhat on google for indications on when this warning
   can be seen. However, I haven't really been able to understand why and
   even if it's significant or not. (At face value I'm nervous the Python
   runtime environment is corrupted for that process).

   2: In my process later on I get: "OverflowError: long too big to convert".
   This happens in different places and seems to always relate to  obtaining
   a length of something (dict or list created by list comprehension). Fx

   "for i in xrange(0, len_of_stuff, max_section_size):"

   en_of_stuff is always less than the max long (around 600).

   We're using gevent and I'm suspecting some "threading" could cause this,
   as I'm able to replicate it locally with the same data.

   Kind regards,

   Thomas

[toc] | [next] | [standalone]

#109061

On Tuesday, May 24, 2016 at 5:47:55 AM UTC-6, thomas povtal.org wrote:
...
>    1: I get "RuntimeWarning: tp_compare didn't return -1 or -2 for
>    exception". It's a line like:
> 
>    "if Foo = False:" where Foo is a global variable (global Foo).
...

Are you really using "if Foo = False:"?
If so, it should be "if Foo == False:"
"==" for equivalence rather than "=" for assignment.

[toc] | [prev] | [next] | [standalone]

#109063

   Hi!

   Thanks! That was a typo. I do '=='.

   The rest of the mail is ok (with the corrections from Chris)

   :) T

     Den 24. maj 2016 klokken 17:05 skrev Novocastrian_Nomad
     <gregory.j.baker@gmail.com>:

     On Tuesday, May 24, 2016 at 5:47:55 AM UTC-6, thomas povtal.org wrote:
     ...
     > 1: I get "RuntimeWarning: tp_compare didn't return -1 or -2 for
     > exception". It's a line like:
     >
     > "if Foo = False:" where Foo is a global variable (global Foo).
     ...

     Are you really using "if Foo = False:"?
     If so, it should be "if Foo == False:"
     "==" for equivalence rather than "=" for assignment.
     --
     https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#109070

On Tue, 24 May 2016 08:22 pm, thomas povtal.org wrote:

>    Hi,
> 
>    Please excuse me if this is not the right place, but I have some issues
>    with CPython on a NUMA machine.

Do you mean a Non-Uniform Memory Access machine? 

Can you be more specific about the actual machine and OS used?


>    1: I get "RuntimeWarning: tp_compare didn't return -1 or -2 for
>    exception". It's a line like:
> 
>    "if Foo = False:" where Foo is a global variable (global Foo).

What is the type and value of Foo?


>    Now, I've searched somewhat on google for indications on when this
>    warning can be seen. However, I haven't really been able to understand
>    why and even if it's significant or not. (At face value I'm nervous the
>    Python runtime environment is corrupted for that process).

At face value, that RuntimeWarning seems to indicate a bug in the
interpreter.


>    2: In my process later on I get: "OverflowError: long too big to
>    convert".

Can you copy and paste the actual traceback rather than retyping it from
memory? I think you're missing something, namely what the long is being
converted to. The rest of the traceback will help too.


>    This happens in different places and seems to always relate to 
>    obtaining a length of something (dict or list created by list
>    comprehension). Fx
> 
>    "for i in xrange(0, len_of_stuff, max_section_size):"
> 
>    en_of_stuff is always less than the max long (around 600).

What do you mean, "the max long"? Longs do not have a max value. The only
limit is the amount of memory you have.

What about max_section_size? How big is that?


>    We're using gevent and I'm suspecting some "threading" could cause
>    this, as I'm able to replicate it locally with the same data.

Typo: you said later that you are *not* able to replicate it.

You're using a global variable with threaded code? You're a brave (or
foolhardy) man...


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#109072

On Wed, May 25, 2016 at 3:12 AM, Steven D'Aprano <steve@pearwood.info> wrote:
>>    2: In my process later on I get: "OverflowError: long too big to
>>    convert".
>
> Can you copy and paste the actual traceback rather than retyping it from
> memory? I think you're missing something, namely what the long is being
> converted to. The rest of the traceback will help too.
>
>
>>    This happens in different places and seems to always relate to
>>    obtaining a length of something (dict or list created by list
>>    comprehension). Fx
>>
>>    "for i in xrange(0, len_of_stuff, max_section_size):"
>>
>>    en_of_stuff is always less than the max long (around 600).
>
> What do you mean, "the max long"? Longs do not have a max value. The only
> limit is the amount of memory you have.

A Python long doesn't, but an ssize_t does. In certain places, Python
integers get converted into C integers, at which point an over-large
value triggers OverflowError.

>>> xrange(1<<100)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long
>>> xrange(1,500,1<<100)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long

(Note: Doesn't happen with a Py3 range object.)

But I echo the request for a copied and pasted error message.

ChrisA

[toc] | [prev] | [next] | [standalone]

#109112

Hi!

Thanks. Replies in-line
> Den 24. maj 2016 klokken 19:12 skrev Steven D'Aprano <steve@pearwood.info>:
> 
> On Tue, 24 May 2016 08:22 pm, thomas povtal.org wrote:
> 
> > Hi,
> > 
> > Please excuse me if this is not the right place, but I have some issues
> > with CPython on a NUMA machine.
> 
> Do you mean a Non-Uniform Memory Access machine? 
> 
> Can you be more specific about the actual machine and OS used?

It's Ubuntu 12.04 LTS. It appears as a NUMA machine with two nodes. However, I
later learned that it's (probably?) not a real NUMA but just Ubuntu's way of
representing two socktets for CPUs. I'm not very skilled in such low-level
matters, unfortunately.
> > 1: I get "RuntimeWarning: tp_compare didn't return -1 or -2 for
> > exception". It's a line like:
> > 
> > "if Foo = False:" where Foo is a global variable (global Foo).
> 
> What is the type and value of Foo?

global Foo = False
> > Now, I've searched somewhat on google for indications on when this
> > warning can be seen. However, I haven't really been able to understand
> > why and even if it's significant or not. (At face value I'm nervous the
> > Python runtime environment is corrupted for that process).
> 
> At face value, that RuntimeWarning seems to indicate a bug in the
> interpreter.

Ok. I was fearing this.
> > 2: In my process later on I get: "OverflowError: long too big to
> > convert".
> 
> Can you copy and paste the actual traceback rather than retyping it from
> memory? I think you're missing something, namely what the long is being
> converted to. The rest of the traceback will help too.

2016-05-24_08:15:40.84187 File "checkrc.pxd", line 14, in
zmq.core.checkrc._check_rc (zmq/core/socket.c:5932)
2016-05-24_08:15:40.84187 OverflowError: long too big to convert

I agree it looks like cut off, but it isn't...
> > This happens in different places and seems to always relate to 
> > obtaining a length of something (dict or list created by list
> > comprehension). Fx
> > 
> > "for i in xrange(0, len_of_stuff, max_section_size):"
> > 
> > en_of_stuff is always less than the max long (around 600).
> 
> What do you mean, "the max long"? Longs do not have a max value. The only
> limit is the amount of memory you have.
> 
> What about max_section_size? How big is that?

30
> > We're using gevent and I'm suspecting some "threading" could cause
> > this, as I'm able to replicate it locally with the same data.
> 
> Typo: you said later that you are *not* able to replicate it.
> 
> You're using a global variable with threaded code? You're a brave (or
> foolhardy) man...

Yeah, I didn't write the code originally, but that's not an excuse. The
greenlets involved doesn't appear to be mulithreaded in this case, though. Even
if it was, could that miss with the interpreter to give these sporious messages?
> -- 
> Steven
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [next] | [standalone]

#109113

On 25/05/2016 13:04, thomas povtal.org wrote:
> 2016-05-24_08:15:40.84187 File "checkrc.pxd", line 14, in
> zmq.core.checkrc._check_rc (zmq/core/socket.c:5932)
> 2016-05-24_08:15:40.84187 OverflowError: long too big to convert

That exception is arising from ZeroMQ's own code, by the look of it. (Or
perhaps pyzmq).

It's not impossible that it's arising ultimately from Python's
interpreter core, but you'd want to take that up with the ZeroMQ guys
first to see what checks they're doing which might raise that exception.

https://github.com/zeromq

TJG

[toc] | [prev] | [next] | [standalone]

#109115

   Hi!

   Thanks. It was an example... I get the very same exception text (the one
   that appears to be cut off) in this line in our own code:

   if Foo == False:

   (where Foo is global Foo = False.)

   :) T

     Den 25. maj 2016 klokken 14:13 skrev Tim Golden <mail@timgolden.me.uk>:

     On 25/05/2016 13:04, thomas povtal.org wrote:
     > 2016-05-24_08:15:40.84187 File "checkrc.pxd", line 14, in
     > zmq.core.checkrc._check_rc (zmq/core/socket.c:5932)
     > 2016-05-24_08:15:40.84187 OverflowError: long too big to convert

     That exception is arising from ZeroMQ's own code, by the look of it. (Or
     perhaps pyzmq).

     It's not impossible that it's arising ultimately from Python's
     interpreter core, but you'd want to take that up with the ZeroMQ guys
     first to see what checks they're doing which might raise that exception.

     https://github.com/zeromq

     TJG
     --
     https://mail.python.org/mailman/listinfo/python-list

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web