Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #82876 > unrolled thread

question on string object handling in Python 2.7.8

Back to article view | Back to comp.lang.python

Contents

  question on string object handling in Python 2.7.8 Dave Tian <dave.jing.tian@gmail.com> - 2014-12-23 20:28 -0500
    Re: question on string object handling in Python 2.7.8 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-24 22:22 +1100
      Re: question on string object handling in Python 2.7.8 Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-24 09:39 -0700
    Re: question on string object handling in Python 2.7.8 Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-12-25 17:23 +1300
    Re: question on string object handling in Python 2.7.8 Denis McMahon <denismfmcmahon@gmail.com> - 2014-12-25 16:34 +0000

#82876 — question on string object handling in Python 2.7.8

Hi,

There are 2 statements:
A: a = ‘h’
B: b = ‘hh’

According to me understanding, A should be faster as characters would shortcut this 1-byte string ‘h’ without malloc; B should be slower than A as characters does not work for 2-byte string ‘hh’, which triggers the malloc. However, when I put A/B into a big loop and try to measure the performance using cProfile, B seems always faster than A.
Testing code:
for i in range(0, 100000000):
	a = ‘h’ #or b = ‘hh’
Testing cmd: python -m cProfile test.py

So what is wrong here? B has one more malloc than A but is faster than B?

Thanks,
Dave

[toc] | [next] | [standalone]

#82882

Dave Tian wrote:

> Hi,
> 
> There are 2 statements:
> A: a = ‘h’
> B: b = ‘hh’
> 
> According to me understanding, A should be faster as characters would
> shortcut this 1-byte string ‘h’ without malloc; B should be slower than A
> as characters does not work for 2-byte string ‘hh’, which triggers the
> malloc. However, when I put A/B into a big loop and try to measure the
> performance using cProfile, B seems always faster than A. 
>
> Testing code: 
> for i in range(0, 100000000): a = ‘h’ #or b = ‘hh’ 
> Testing cmd: python -m cProfile test.py

Any performance difference is entirely an artifact of your testing method.
You have completely misinterpreted what this piece of code will do.

What happens here is that you time a piece of code to:

- Build a large list containing 100 million individual int objects. Each int
object has to be allocated at run time, as does the list. Each int object
is about 12 bytes in size.

- Then, the name i is bound to one of those int objects. This is a fast
pointer assignment.

- Then, a string object containing either 'h' or 'hh' is allocated. In
either case, that requires 21 bytes, plus one byte per character. So either
22 or 23 bytes.

- The name a is bound to that string object. This is also a fast pointer
assignment.

- The loop returns to the top, and the name i is bound to the next int
object.

- Then, the name a is bound *to the same string object*, since it will have
been cached. No further malloc will be needed.


So as you can see, the time you measure is dominated by allocating a massive
list containing 100 million int objects. Only a single string object is
allocated, and the time difference between creating 'h' versus 'hh' is
insignificant.

The byte code can be inspected like this:

py> code = compile("for i in range(100000000): a = 'h'", '', 'exec')
py> from dis import dis
py> dis(code)
  1           0 SETUP_LOOP              26 (to 29)
              3 LOAD_NAME                0 (range)
              6 LOAD_CONST               0 (100000000)
              9 CALL_FUNCTION            1
             12 GET_ITER
        >>   13 FOR_ITER                12 (to 28)
             16 STORE_NAME               1 (i)
             19 LOAD_CONST               1 ('h')
             22 STORE_NAME               2 (a)
             25 JUMP_ABSOLUTE           13
        >>   28 POP_BLOCK
        >>   29 LOAD_CONST               2 (None)
             32 RETURN_VALUE


Notice instruction 19 and 22:

             19 LOAD_CONST               1 ('h')
             22 STORE_NAME               2 (a)

The string object is built at compile time, not run time, and Python simply
binds the name a to the pre-existing string object.

If you looked at the output of your timing code, you would see something
like this (only the times would be much larger, I cut the loop down from
100 million to only a few tens of thousands):


[steve@ando ~]$ python -m cProfile /tmp/x.py
         3 function calls in 0.062 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.051    0.051    0.062    0.062 x.py:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable'
of '_lsprof.Profiler' objects}
        1    0.012    0.012    0.012    0.012 {range}


The profiler doesn't even show the time required to bind the name to the
string object.

Here is a better way of demonstrating the same thing:


py> from timeit import Timer
py> t = Timer("a = 'h'")
py> min(t.repeat())
0.0508120059967041
py> t = Timer("a = 'hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh'")
py> min(t.repeat())
0.050585031509399414

No meaningful difference in time. What difference you do see is a fluke of
timing.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#82899

[Multipart message — attachments visible in raw view] — view raw

On Wed, Dec 24, 2014 at 4:22 AM, Steven D'Aprano <
steve+comp.lang.python@pearwood.info> wrote:
> What happens here is that you time a piece of code to:
>
> - Build a large list containing 100 million individual int objects. Each
int
> object has to be allocated at run time, as does the list. Each int object
> is about 12 bytes in size.

Note to the OP: since you're using Python 2 you would do better to loop
over an xrange object instead of a range. xrange produces an iterator over
the desired range without needing to construct a single list containing all
of them. They would all still need to be allocated, but not all at once,
and memory could be reused.

[toc] | [prev] | [next] | [standalone]

#82909

Dave Tian wrote:

> A: a = ‘h’ 
 > B: b = ‘hh’
> 
> According to me understanding, A should be faster as characters would
> shortcut this 1-byte string ‘h’ without malloc;

It sounds like you're expecting characters to be stored
"unboxed" like in Java.

That's not the way Python works. Objects are used for
everything, including numbers and characters (there is
no separate character type in Python, they're just
length-1 strings).

 > for i in range(0, 100000000):
 >	a = ‘h’ #or b = ‘hh’
 > Testing cmd: python -m cProfile test.py

Since you're assigning a string literal, there's just
one string object being allocated (at the time the code
is read in and compiled). All the loop is doing is
repeatedly assigning a reference to that object to a
or b, which doesn't require any further mallocs;
all it does is adjust reference counts. This will
be swamped by the overhead of the for-loop itself,
which is allocating and deallocating 100 million
integer objects.

I would expect both of these to be exactly the same
speed, within measurement error. Any difference you're
seeing is probably just noise, or the result of some
kind of warming-up effect.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#82923

On Tue, 23 Dec 2014 20:28:30 -0500, Dave Tian wrote:

> Hi,
> 
> There are 2 statements:
> A: a = ‘h’
> B: b = ‘hh’
> 
> According to me understanding, A should be faster as characters would
> shortcut this 1-byte string ‘h’ without malloc; B should be slower than
> A as characters does not work for 2-byte string ‘hh’, which triggers the
> malloc. However, when I put A/B into a big loop and try to measure the
> performance using cProfile, B seems always faster than A.
> Testing code:
> for i in range(0, 100000000):
> 	a = ‘h’ #or b = ‘hh’
> Testing cmd: python -m cProfile test.py
> 
> So what is wrong here? B has one more malloc than A but is faster than
> B?

Your understanding.

The first time through the loop, python creates a string object "h" or 
"hh" and creates a pointer (a or b) and assigns it to the string object.

The next range(1, 100000000) times through the loop, python re-assigns 
the existing pointer to the existing string object.

Maybe a 2 character string is faster to locate in the object table than a 
1 character string, so that in the 2 character case, the lookup is faster.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web