Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Ben Finney <ben+python@benfinney.id.au>
Newsgroups: comp.lang.python
Subject: Re: Storing a big amount of path names
Date: Fri, 12 Feb 2016 11:44:17 +1100
Lines: 31
Message-ID: <mailman.61.1455237865.22075.python-list@python.org>
References: <n9j94f$712$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de vE6/AFmP57r0KxhDLrnaDgtjbgpz1NP85erbWpuGQL5w==
Cancel-Lock: sha1:LA7ehIVW90aa/WyuHSPKnGpud9o=
Return-Path: <python-python-list@m.gmane.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.022
X-Spam-Evidence: '*H*': 0.96; '*S*': 0.00; 'here?': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'python': 0.10; 'suggest': 0.15; '(shortest': 0.16; 'measured': 0.16; 'optimised': 0.16; 'pathnames': 0.16; 'paulo': 0.16; 'received:80.91.229.3': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'suggestion.': 0.16; 'memory': 0.17; 'amounts': 0.22; "python's": 0.23; 'header:User-Agent:1': 0.26; 'header:X-Complaints-To:1': 0.26; 'container': 0.29; 'true.': 0.33; 'confirmed': 0.35; 'primarily': 0.35; 'quite': 0.35; 'expected': 0.35; 'but': 0.36; 'too': 0.36; 'should': 0.36; 'data.': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'thanks': 0.37; 'received:org': 0.37; 'why': 0.39; 'data': 0.39; 'to:addr:python.org': 0.40; 'your': 0.60; 'challenge': 0.61; 'skip:u 10': 0.61; 'our': 0.64; 'quantity': 0.66; '_o__)': 0.84; 'received:125': 0.84; '\xe2\x80\x9cwe': 0.84; 'items,': 0.91; 'suffer': 0.93; 'factors': 0.97
X-Injected-Via-Gmane: http://gmane.org/
X-Gmane-NNTP-Posting-Host: jigong.madmonks.org
X-Public-Key-ID: 0xAC128405
X-Public-Key-Fingerprint: 517C F14B B2F3 98B0 CB35  4855 B8B2 4C06 AC12 8405
X-Public-Key-URL: http://www.benfinney.id.au/contact/bfinney-pubkey.asc
X-Post-From: Ben Finney <bignose+hates-spam@benfinney.id.au>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.21rc2
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:102835

Paulo da Silva <p_s_d_a_s_i_l_v_a_ns@netcabo.pt> writes:

> What is the best (shortest memory usage) way to store lots of
> pathnames in memory

I challenge the premise. Why is “shortest memory usage” your criterion
for “best”, here?

How have you determined that factors like “easily understandable when
reading”, or “using standard Python idioms”, are less important?

As for “lots of pathnames”, how many are you expecting? Python's
built-in container types are highly optimised for quite large amounts of
data.

Have you measured an implementation with normal built-in container types
with your expected quantity of items, and confirmed that the performance
is unacceptable?

> Thanks for any suggestion.

I would suggest that the assumption you have too much data for Python's
built-in container types, is an assumption that should be rigorously
tested because it is likely not true.

-- 
 \      “We suffer primarily not from our vices or our weaknesses, but |
  `\    from our illusions.” —Daniel J. Boorstin, historian, 1914–2004 |
_o__)                                                                  |
Ben Finney