Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Chris Angelico <rosuav@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: How to memory dump an object?
Date: Sat, 21 May 2016 12:28:40 +1000
Lines: 56
Message-ID: <mailman.66.1463797722.27390.python-list@python.org>
References: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> <CAPTjJmrjiXneriVNRZ-Ldp74YQJ4dmu_rKAMYz_ZQ0WzVuOo3Q@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com>
Precedence: list
Xref: csiph.com comp.lang.python:108886

On Sat, May 21, 2016 at 11:05 AM,  <jfong@ms4.hinet.net> wrote:
> Is there any tools which can do the memory dump of an object so I can vie=
w their content or implementation? For example,
>
>>>> s1 =3D '\x80abc'
>>>> b1 =3D b'\x80abc'
>
> What are exactly stored in memory for each of them? Is their content real=
ly the same? This kind of tool should be helpful "for me" to learn the inne=
r detail of Python.
>

MASSIVE CAVEAT: This is *not* anything that the Python language
specifies. You can mess around with this in one interpreter (say,
CPython 3.5) and get one result, but in a different interpreter
(Jython, IronPython, MicroPython, Brython), or a different version of
the same interpreter (CPython 3.2), or even the exact same interpreter
on different platforms (32-bit vs 64-bit, Linux vs Windows vs Mac OS
vs other, or different CPUs), you may get completely different
results. And some language implementations (eg PyPy) don't necessarily
even keep the object in memory at all times.

But if you're curious about how things are stored, there are ways you
can mess around and find stuff out.

The easiest way to explore CPython's internals would probably be
ctypes. In CPython, an object's identity is based on its address in
memory, so you can cast that to a pointer.

>>> import ctypes
>>> s1 =3D "\x80abc"
>>> ctypes.cast(id(s1), ctypes.c_voidp)
c_void_p(140180250548144)
>>> import sys
>>> sys.getsizeof(s1) # See how much space it takes up in RAM
77

You can mess around with pointers to your heart's content:

>>> ptr =3D ctypes.cast(id(s1), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([ptr[i] for i in range(77)])
b'\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x88\x00\x00\x00\x00\x00\x04\x00\=
x00\x00\x00\x00\x00\x00\x15!\xbc\x99y\xc4A]\xa43XB~\x7f\x00\x00\x00\x00\x00=
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0=
0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00'

Hey look, there it is!

For more info on how CPython 3.3+ stores Unicode text, check out this docum=
ent:

https://www.python.org/dev/peps/pep-0393/

It's pretty smart :)

ChrisA