Re: How to memory dump an object?

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Chris Angelico <rosuav@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: How to memory dump an object?
Date	Sat, 21 May 2016 12:28:40 +1000
Lines	56
Message-ID	<mailman.66.1463797722.27390.python-list@python.org> (permalink)
References	<6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com> <CAPTjJmrjiXneriVNRZ-Ldp74YQJ4dmu_rKAMYz_ZQ0WzVuOo3Q@mail.gmail.com>
Mime-Version	1.0
Content-Type	text/plain; charset=UTF-8
Content-Transfer-Encoding	quoted-printable
X-Trace	news.uni-berlin.de KHTob9DtqoYaY1QEDFkl5Qi85RJNPoH+DhnlZwSAv7qQ==
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.000
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'cpython': 0.05; 'sys': 0.05; 'not': 0.07; '21,': 0.07; 'cc:addr:python-list': 0.09; 'subject:How': 0.09; 'content:': 0.09; 'mess': 0.09; "object's": 0.09; 'pointers': 0.09; 'stored': 0.10; 'python': 0.10; 'python.': 0.11; 'interpreter': 0.15; '2016': 0.16; '64-bit,': 0.16; 'ctypes.': 0.16; 'document:': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'ptr': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'subject:object': 0.16; 'wrote:': 0.16; 'memory': 0.17; 'result,': 0.18; 'platforms': 0.18; 'language': 0.19; '>>>': 0.20; 'hey': 0.20; 'windows': 0.20; 'cc:2**0': 0.20; 'cc:addr:python.org': 0.20; 'ctypes': 0.22; 'text,': 0.22; 'am,': 0.23; 'sat,': 0.23; 'import': 0.24; 'header :In-Reply-To:1': 0.24; 'linux': 0.26; 'helpful': 0.27; 'message- id:@mail.gmail.com': 0.27; 'other,': 0.29; 'pointer.': 0.29; 'them?': 0.29; 'url:peps': 0.29; 'probably': 0.31; 'url:python': 0.33; 'curious': 0.33; 'ram': 0.33; 'smart': 0.33; 'info': 0.34; 'received:google.com': 0.35; 'skip:c 30': 0.35; 'easiest': 0.35; 'identity': 0.35; 'unicode': 0.35; 'url:dev': 0.35; 'received:74.125.82': 0.35; 'but': 0.36; 'should': 0.36; 'there': 0.36; 'url:org': 0.36; 'tool': 0.36; 'subject:?': 0.36; 'subject:: ': 0.37; 'really': 0.37; 'mac': 0.37; 'things': 0.38; 'detail': 0.38; 'times.': 0.38; 'version': 0.38; 'stuff': 0.38; 'anything': 0.38; 'takes': 0.39; 'space': 0.40; 'some': 0.40; 'your': 0.60; 'address': 0.61; 'more': 0.63; 'different': 0.63; 'necessarily': 0.63; 'cast': 0.66; 'results.': 0.67; 'chrisa': 0.84; 'is!': 0.84; 'stored,': 0.84; 'to:none': 0.91
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-transfer-encoding; bh=UVU8e7qLIjceOd0MMS0KIjhqI5Itcxhu/8n+QEi6oyk=; b=Y17bW4EAjIG5sctej/aC+ORR98NhVUYuffLhGP9i0UNCT0MgOUPVpV2Ss9nh9llO/k +9+k2g0aEfoIil/lx/ynuK+G3YBdwEDis2kZuV7gdPqJwJkzBUlbuKMVDnI9629pBda2 ljNPVlsXgduHLuWFlCsjl4Qty8f1UrE2bjXleipoNCb23GCdgBLy1fQS11n/ZIY8IGys nuaRfuho63FF38xIsY/W3LQb5yMN5MS2f5wlhvJyj8XfeZUFevZxZTMAwNmQG1kYe2ZG JWH6yNP1FNUxD3RSwvSYh7q9V92GgCkAagtopOQDUOdNXq+Qk73qXXK5Tva6ZgNl8shI g87Q==
X-Google-DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:cc:content-transfer-encoding; bh=UVU8e7qLIjceOd0MMS0KIjhqI5Itcxhu/8n+QEi6oyk=; b=PH6Ij9GYdJTmKiVFh4SgBG2U03sbgv8HuhYGD8prNznlldtYIyd1c2oxG8yPXDsHLe LoNKSUtSZ7/K019s6G4jyS8RKU1QvMuEQNGdY2fsZwSvHLq7wF/CyTQ5RA1nu7LTjPC4 fiklstsGTUBBz9q/Tkzx8WKrlj4RAw1Pq8+O+wirnUDH65j2NgKDf6tESMaxmX+LwbHa kfDgDLr771aS81u3w2W29uO8X9jNii6TRUiUFNVL8LjH1VsZOy3avHjaF6RvZGcdSvu2 +EtQJV8oIE7WL+SvKt7KgDpiYH6oV/nrt+efOJqq1H6fjb6vue2YjKaeji3rMTgkf0Hh XokQ==
X-Gm-Message-State	AOPr4FUAJRG5R+FgSlBUc99fyEIqpP2bNp+vOXQhNcjVYMkDN3ZZTEsVCmG7Oqeg5h/uV17Csy74AV02YDRF4w==
X-Received	by 10.194.86.129 with SMTP id p1mr5882469wjz.117.1463797720749; Fri, 20 May 2016 19:28:40 -0700 (PDT)
In-Reply-To	<6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.22
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID	<CAPTjJmrjiXneriVNRZ-Ldp74YQJ4dmu_rKAMYz_ZQ0WzVuOo3Q@mail.gmail.com>
X-Mailman-Original-References	<6c556e26-1350-4fdc-ae9f-7ef198a3ad6b@googlegroups.com>
Xref	csiph.com comp.lang.python:108886

Show key headers only | View raw

On Sat, May 21, 2016 at 11:05 AM,  <jfong@ms4.hinet.net> wrote:
> Is there any tools which can do the memory dump of an object so I can view their content or implementation? For example,
>
>>>> s1 = '\x80abc'
>>>> b1 = b'\x80abc'
>
> What are exactly stored in memory for each of them? Is their content really the same? This kind of tool should be helpful "for me" to learn the inner detail of Python.
>

MASSIVE CAVEAT: This is *not* anything that the Python language
specifies. You can mess around with this in one interpreter (say,
CPython 3.5) and get one result, but in a different interpreter
(Jython, IronPython, MicroPython, Brython), or a different version of
the same interpreter (CPython 3.2), or even the exact same interpreter
on different platforms (32-bit vs 64-bit, Linux vs Windows vs Mac OS
vs other, or different CPUs), you may get completely different
results. And some language implementations (eg PyPy) don't necessarily
even keep the object in memory at all times.

But if you're curious about how things are stored, there are ways you
can mess around and find stuff out.

The easiest way to explore CPython's internals would probably be
ctypes. In CPython, an object's identity is based on its address in
memory, so you can cast that to a pointer.

>>> import ctypes
>>> s1 = "\x80abc"
>>> ctypes.cast(id(s1), ctypes.c_voidp)
c_void_p(140180250548144)
>>> import sys
>>> sys.getsizeof(s1) # See how much space it takes up in RAM
77

You can mess around with pointers to your heart's content:

>>> ptr = ctypes.cast(id(s1), ctypes.POINTER(ctypes.c_uint8))
>>> bytes([ptr[i] for i in range(77)])
b'\x01\x00\x00\x00\x00\x00\x00\x00\x00\x04\x88\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x15!\xbc\x99y\xc4A]\xa43XB~\x7f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80abc\x00'

Hey look, there it is!

For more info on how CPython 3.3+ stores Unicode text, check out this document:

https://www.python.org/dev/peps/pep-0393/

It's pretty smart :)

ChrisA

Thread

How to memory dump an object? jfong@ms4.hinet.net - 2016-05-20 18:05 -0700
  Re: How to memory dump an object? Ned Batchelder <ned@nedbatchelder.com> - 2016-05-20 18:59 -0700
  Re: How to memory dump an object? Chris Angelico <rosuav@gmail.com> - 2016-05-21 12:28 +1000
    Re: How to memory dump an object? jfong@ms4.hinet.net - 2016-05-20 21:47 -0700
      Re: How to memory dump an object? Chris Angelico <rosuav@gmail.com> - 2016-05-21 15:01 +1000
    Re: How to memory dump an object? wxjmfauth@gmail.com - 2016-05-20 23:55 -0700
  Re: How to memory dump an object? Steven D'Aprano <steve@pearwood.info> - 2016-05-21 14:56 +1000
  Re: How to memory dump an object? jfong@ms4.hinet.net - 2016-05-21 01:08 -0700

csiph-web