Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #58944

Re: chunking a long string?

Show key headers only | View raw

On Sun, Nov 10, 2013 at 1:37 AM, Roy Smith <roy@panix.com> wrote:
> In article <mailman.2283.1383985583.18130.python-list@python.org>,
>  Chris Angelico <rosuav@gmail.com> wrote:
>
>> Some languages [intern] automatically for all strings, others
>> (like Python) only when you ask for it.
>
> What does "only when you ask for it" mean?

You can explicitly intern a Python string with the sys.intern()
function, which returns either the string itself or an
indistinguishable "interned" string. Two equal strings, when interned,
will return the same object:

>>> foo = "asdf"
>>> bar = "as"
>>> bar += "df"
>>> foo is bar
False

Note that the Python interpreter is free to answer True there, but
there's no mandate for it.

>>> foo = sys.intern(foo)
>>> bar = sys.intern(bar)
>>> foo is bar
True

Now it's mandated. The two strings must be the same object. Interning
in this way makes string equality come down to an 'is' check, which is
potentially a lot faster than actual string equality.

Some languages (eg Pike) do this automatically with all strings - the
construction of any string includes checking to see if it's a
duplicate of any other string. This adds cost to string manipulation
and speeds up string comparisons; since the engine knows that all
strings are interned, it can do the equivalent of an 'is' check for
any string equality.

So what I meant, in terms of storage/representation efficiency, is
that you can store duplicate strings very efficiently if you simply
increment the reference counts of the same few objects. Python won't
necessarily do that for you; check memory usage of something like
this:

strings = [open("some_big_file").read() for _ in range(10000)]

And compare against this:

strings = [sys.intern(open("some_big_file").read()) for _ in range(10000)]

In a language that guarantees string interning, the syntax of the
former would have the memory consumption of the latter. Whether that
memory saving and improved equality comparison is worth the effort of
dictionarification is one of those eternally-debatable points.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

chunking a long string? Roy Smith <roy@panix.com> - 2013-11-08 12:48 -0500
  Re: chunking a long string? wxjmfauth@gmail.com - 2013-11-08 12:43 -0800
    Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-09 07:53 +1100
    Re: chunking a long string? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-11-08 20:57 +0000
    Re: chunking a long string? Tim Chase <python.list@tim.thechases.com> - 2013-11-08 15:04 -0600
    Re: chunking a long string? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-11-08 21:06 +0000
    Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-09 08:04 +1100
    Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-09 08:17 +1100
    Re: chunking a long string? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-11-09 00:46 +0000
      Re: chunking a long string? wxjmfauth@gmail.com - 2013-11-09 00:14 -0800
        Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-09 19:26 +1100
          Re: chunking a long string? Roy Smith <roy@panix.com> - 2013-11-09 09:37 -0500
            Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-10 02:02 +1100
              Re: chunking a long string? Roy Smith <roy@panix.com> - 2013-11-09 10:21 -0500
                Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-10 02:30 +1100
                Re: chunking a long string? Roy Smith <roy@panix.com> - 2013-11-09 10:35 -0500
            Re: chunking a long string? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-11-09 15:37 +0000
              Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-10 09:14 +1100
                Re: chunking a long string? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-11-10 06:39 +0000
                Re: chunking a long string? Chris Angelico <rosuav@gmail.com> - 2013-11-10 19:46 +1100
        Re: chunking a long string? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-11-09 10:13 +0000
  Re: chunking a long string? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-11-09 00:54 +0000

csiph-web