Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #108880 > unrolled thread

How to memory dump an object?

Back to article view | Back to comp.lang.python

Contents

  How to memory dump an object? jfong@ms4.hinet.net - 2016-05-20 18:05 -0700
    Re: How to memory dump an object? Ned Batchelder <ned@nedbatchelder.com> - 2016-05-20 18:59 -0700
    Re: How to memory dump an object? Chris Angelico <rosuav@gmail.com> - 2016-05-21 12:28 +1000
      Re: How to memory dump an object? jfong@ms4.hinet.net - 2016-05-20 21:47 -0700
        Re: How to memory dump an object? Chris Angelico <rosuav@gmail.com> - 2016-05-21 15:01 +1000
      Re: How to memory dump an object? wxjmfauth@gmail.com - 2016-05-20 23:55 -0700
    Re: How to memory dump an object? Steven D'Aprano <steve@pearwood.info> - 2016-05-21 14:56 +1000
    Re: How to memory dump an object? jfong@ms4.hinet.net - 2016-05-21 01:08 -0700

#108880 — How to memory dump an object?

Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example,

>>> s1 = '\x80abc'
>>> b1 = b'\x80abc'

What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python. 


--Jach

[toc] | [next] | [standalone]

#108882

On Friday, May 20, 2016 at 9:05:51 PM UTC-4, jf...@ms4.hinet.net wrote:
> Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example,
> 
> >>> s1 = '\x80abc'
> >>> b1 = b'\x80abc'
> 
> What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python. 

I don't know of a tool that will do that, other than running CPython under
the gdb debugger, and examining memory that way.

In Python 2, those two objects are the same, because '...' is a byte string,
and b'...' is a byte string.

I should say, those objects' memory starts out exactly the same.  Objects have
reference counts which change as names come and go:

    >>> s1 = '\x80abc'
    >>> b1 = b'\x80abc'
    >>> b2 = b1

Now the first string has a reference count of 1, and b1 has a reference count
of 2. Those counts are in the objects' memory, so now their memory contents
are different.

--Ned.

[toc] | [prev] | [next] | [standalone]

#108886

On Sat, May 21, 2016 at 11:05 AM,  <jfong@ms4.hinet.net> wrote:
> Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example,
>
>>>> s1 = '\x80abc'
>>>> b1 = b'\x80abc'
>
> What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python.
>

MASSIVE CAVEAT: This is *not* anything that the Python language
specifies. You can mess around with this in one interpreter (say,
CPython 3.5) and get one result, but in a different interpreter
(Jython, IronPython, MicroPython, Brython), or a different version of
the same interpreter (CPython 3.2), or even the exact same interpreter
on different platforms (32-bit vs 64-bit, Linux vs Windows vs Mac OS
vs other, or different CPUs), you may get completely different
results. And some language implementations (eg PyPy) don't necessarily
even keep the object in memory at all times.

But if you're curious about how things are stored, there are ways you
can mess around and find stuff out.

The easiest way to explore CPython's internals would probably be
ctypes. In CPython, an object's identity is based on its address in
memory, so you can cast that to a pointer.

>>> import ctypes
>>> s1 = "\x80abc"
>>> ctypes.cast(id(s1), ctypes.c_voidp)
c_void_p(140180250548144)
>>> import sys
>>> sys.getsizeof(s1) # See how much space it takes up in RAM
77

You can mess around with pointers to your heart's content:

>>> ptr = ctypes.cast(id(s1), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([ptr[i] for i in range(77)])
b'\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x88\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x15!\xbc\x99y\xc4A]\xa43XB~\x7f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00'

Hey look, there it is!

For more info on how CPython 3.3+ stores Unicode text, check out this document:

https://www.python.org/dev/peps/pep-0393/

It's pretty smart :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#108893

Sorry, forget to mention that I am working on version 3.4

Following the steps given in Chris's reply, I get the result from bytes string:

>>> b1 = b'\x80abc'
>>> ctypes.cast(id(b1), ctypes.c_voidp)
c_void_p(35495992)
>>> sys.getsizeof(b1)
21
>>> b1ptr = ctypes.cast(id(b1), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([b1ptr[i] for i in range(21)])
b'\x01\x00\x00\x00X\xa1e^\x04\x00\x00\x00\x04\xff\x04&\x80abc\x00'
>>>

It obviously has the same "content" as the s1 string has and the object's structure are much different as expected (77 vs 21 bytes:-)

If I do this as Ned suggested:

>>> b2 = b1

I get:

>>> bytes([b1ptr[i] for i in range(21)])
b'\x02\x00\x00\x00X\xa1e^\x04\x00\x00\x00\x04\xff\x04&\x80abc\x00'
>>>

So, at least, we know where the reference count was stored:-)

It's amazing the "tool" is so simple. Actually I feel a little embarrassed that I do know every statements in it but just no idea of combining them together. Thanks a lot, Chris.

--Jach

[toc] | [prev] | [next] | [standalone]

#108895

On Sat, May 21, 2016 at 2:47 PM,  <jfong@ms4.hinet.net> wrote:
> It's amazing the "tool" is so simple. Actually I feel a little embarrassed that I do know every statements in it but just no idea of combining them together. Thanks a lot, Chris.
>

It's a pretty complicated tool, actually - but you're using it in a
simple way here. If you're curious, you can poke around in the CPython
sources to track down the data structures involved - as you noted, the
refcount is at the beginning of every object, and after that you'll
find some neat info like a pointer to its type. You can even change
some of this stuff, but be aware that you can EASILY segfault Python
like this :D

ChrisA

[toc] | [prev] | [next] | [standalone]

#108899

Le samedi 21 mai 2016 04:28:55 UTC+2, Chris Angelico a écrit :
> >
> 
> 
> For more info on how CPython 3.3+ stores Unicode text, check out this document:
> 
> https://www.python.org/dev/peps/pep-0393/
> 
> It's pretty smart :)
> 

A jewel. It does the opposite of "utf-8" when it
comes to memory and the opposite of "utf-32" on
the side of performance...

(And it's buggy).

[toc] | [prev] | [next] | [standalone]

#108894

On Sat, 21 May 2016 11:05 am, jfong@ms4.hinet.net wrote:

> Is there any tools which can do the memory dump of an object so I can view
> their content or implementation? 

No standard tool. There may be third-party tools, but they would be
implementation-specific.


> For example, 
> 
>>>> s1 = '\x80abc'
>>>> b1 = b'\x80abc'
> 
> What are exactly stored in memory for each of them? 

That will depend on the version and implementation of the Python interpreter
you are looking at.

Strings especially are implemented in many different ways. Before Python
3.3, CPython used two different implementations, which you choose when
compiling the interpreter from source:

- "narrow builds" use two-bytes per character, and have difficulty 
  with Unicode codepoints after U+FFFF;
- "wide builds" use four-bytes per character, and can cope with 
  the entire Unicode range up to U+10FFFF;
- I think that Jython uses the Java string implementation;
- and that IronPython uses the .Net string implementation.

Starting at Python 3.3, the CPython implementation will dynamically choose a
one-byte, two-byte or four-byte per character implementation, as needed.


> Is their content really the same?

The first is the Unicode string with four code points:

<control-0080>
LATIN SMALL LETTER A
LATIN SMALL LETTER B
LATIN SMALL LETTER C

the other contains a byte-string with four bytes:

\x80
\x61 (ASCII 'a')
\x62 (ASCII 'b')
\x63 (ASCII 'c')

So in *some* sense they are the same content, but only in the sense that
this list:

[128, 97, 98, 99]

*also* has the same content.


Instead of a low-level memory dump, which isn't actually terribly useful
(for many objects, such as lists, all it would show is a bunch of
pointers), there are other high-level introspection tools available in the
`inspect` module. You can also use:

sys.getsizeof

to see how much memory is used by an object.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#108902

Make a quick experiment under version 3.4.4 through this simple "tool" Chris had provided, now I know how the unicode string was stored in memory:-)

>>> s1 = '\x80abc'
>>> s1
'\x80abc'
>>> len(s1)
4
>>> sys.getsizeof(s1)
41
>>> s1ptr = ctypes.cast(id(s1), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([s1ptr[i] for i in range(41)])
b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x00\x04\xff\x04&\xa4\x00\x00d\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00'
>>>
# a simple unicode string, each code point can be stored in one byte

>>> s2 = '\u0080abc'
>>> s2
'\x80abc'
>>> len(s2)
4
>>> sys.getsizeof(s2)
41
>>> s2ptr = ctypes.cast(id(s2), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([s2ptr[i] for i in range(41)])
b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x00\x04\xff\x04&\xa4\x00\x00|\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00'
>>>
# Python change it automatically to store each code point into one byte

>>> s3 = '\u07b4abc'
>>> s3
'\u07b4abc'
>>> len(s3)
4
>>> sys.getsizeof(s3)
46
>>> s3ptr = ctypes.cast(id(s3), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([s3ptr[i] for i in range(46)])
b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x00\x19n\x93]\xa8\x00\x00dd\r\x1d\x02\x
00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\xb4\x07a\x00b\x00c\x00\x00\x00'
>>>
# use two bytes to store each code point, even the end-of-string mark

>>> s4 = '\U00025049abc'
>>> s4
'\U00025049abc'
>>> len(s4)
4
>>> sys.getsizeof(s4)
56
>>> s4ptr = ctypes.cast(id(s4), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([s4ptr[i] for i in range(56)])
b'\x01\x00\x00\x00\xe8\xa5g^\x04\x00\x00\x003\xdd\xa7"\xb0\x00\x00\x00\x00\x00\x
00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00IP\x02\x00a\x00\x00\x00b\x
00\x00\x00c\x00\x00\x00\x00\x00\x00\x00'
>>>
# use four bytes to store each code point

--Jach

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web