Re: Storing a big amount of path names

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Ben Finney <ben+python@benfinney.id.au>
Newsgroups	comp.lang.python
Subject	Re: Storing a big amount of path names
Date	Fri, 12 Feb 2016 11:44:17 +1100
Lines	31
Message-ID	<mailman.61.1455237865.22075.python-list@python.org> (permalink)
References	<n9j94f$712$1@gioia.aioe.org>
Mime-Version	1.0
Content-Type	text/plain; charset=utf-8
Content-Transfer-Encoding	8bit
X-Trace	news.uni-berlin.de vE6/AFmP57r0KxhDLrnaDgtjbgpz1NP85erbWpuGQL5w==
Cancel-Lock	sha1:LA7ehIVW90aa/WyuHSPKnGpud9o=
Return-Path	<python-python-list@m.gmane.org>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.022
X-Spam-Evidence	'H': 0.96; 'S': 0.00; 'here?': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'python': 0.10; 'suggest': 0.15; '(shortest': 0.16; 'measured': 0.16; 'optimised': 0.16; 'pathnames': 0.16; 'paulo': 0.16; 'received:80.91.229.3': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'suggestion.': 0.16; 'memory': 0.17; 'amounts': 0.22; "python's": 0.23; 'header:User-Agent:1': 0.26; 'header:X-Complaints-To:1': 0.26; 'container': 0.29; 'true.': 0.33; 'confirmed': 0.35; 'primarily': 0.35; 'quite': 0.35; 'expected': 0.35; 'but': 0.36; 'too': 0.36; 'should': 0.36; 'data.': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'thanks': 0.37; 'received:org': 0.37; 'why': 0.39; 'data': 0.39; 'to:addr:python.org': 0.40; 'your': 0.60; 'challenge': 0.61; 'skip:u 10': 0.61; 'our': 0.64; 'quantity': 0.66; '_o__)': 0.84; 'received:125': 0.84; '\xe2\x80\x9cwe': 0.84; 'items,': 0.91; 'suffer': 0.93; 'factors': 0.97
X-Injected-Via-Gmane	http://gmane.org/
X-Gmane-NNTP-Posting-Host	jigong.madmonks.org
X-Public-Key-ID	0xAC128405
X-Public-Key-Fingerprint	517C F14B B2F3 98B0 CB35 4855 B8B2 4C06 AC12 8405
X-Public-Key-URL	http://www.benfinney.id.au/contact/bfinney-pubkey.asc
X-Post-From	Ben Finney <bignose+hates-spam@benfinney.id.au>
User-Agent	Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux)
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.21rc2
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:102835

Show key headers only | View raw

Paulo da Silva <p_s_d_a_s_i_l_v_a_ns@netcabo.pt> writes:

> What is the best (shortest memory usage) way to store lots of
> pathnames in memory

I challenge the premise. Why is “shortest memory usage” your criterion
for “best”, here?

How have you determined that factors like “easily understandable when
reading”, or “using standard Python idioms”, are less important?

As for “lots of pathnames”, how many are you expecting? Python's
built-in container types are highly optimised for quite large amounts of
data.

Have you measured an implementation with normal built-in container types
with your expected quantity of items, and confirmed that the performance
is unacceptable?

> Thanks for any suggestion.

I would suggest that the assumption you have too much data for Python's
built-in container types, is an assumption that should be rigorously
tested because it is likely not true.

-- 
 \      “We suffer primarily not from our vices or our weaknesses, but |
  `\    from our illusions.” —Daniel J. Boorstin, historian, 1914–2004 |
_o__)                                                                  |
Ben Finney

Thread

Storing a big amount of path names Paulo da Silva <p_s_d_a_s_i_l_v_a_ns@netcabo.pt> - 2016-02-12 00:31 +0000
  Re: Storing a big amount of path names Chris Angelico <rosuav@gmail.com> - 2016-02-12 11:39 +1100
  Re: Storing a big amount of path names Ben Finney <ben+python@benfinney.id.au> - 2016-02-12 11:44 +1100
  Re: Storing a big amount of path names Tim Chase <python.list@tim.thechases.com> - 2016-02-11 19:13 -0600
    Re: Storing a big amount of path names Rob Gaddi <rgaddi@highlandtechnology.invalid> - 2016-02-12 02:17 +0000
  Re: Storing a big amount of path names MRAB <python@mrabarnett.plus.com> - 2016-02-12 03:13 +0000
  Re: Storing a big amount of path names Chris Angelico <rosuav@gmail.com> - 2016-02-12 14:49 +1100
    Re: Storing a big amount of path names Paulo da Silva <p_s_d_a_s_i_l_v_a_ns@netcabo.pt> - 2016-02-12 04:15 +0000
      Re: Storing a big amount of path names Chris Angelico <rosuav@gmail.com> - 2016-02-12 15:23 +1100
        Re: Storing a big amount of path names Paulo da Silva <p_s_d_a_s_i_l_v_a_ns@netcabo.pt> - 2016-02-12 04:45 +0000
          Re: Storing a big amount of path names Chris Angelico <rosuav@gmail.com> - 2016-02-12 16:02 +1100
            Re: Storing a big amount of path names Paulo da Silva <p_s_d_a_s_i_l_v_a_ns@netcabo.pt> - 2016-02-12 05:49 +0000
            Re: Storing a big amount of path names Steven D'Aprano <steve@pearwood.info> - 2016-02-12 16:51 +1100
        Re: Storing a big amount of path names Rob Gaddi <rgaddi@highlandtechnology.invalid> - 2016-02-12 17:05 +0000
          Re: Storing a big amount of path names Chris Angelico <rosuav@gmail.com> - 2016-02-13 04:18 +1100
          Re: Storing a big amount of path names Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-02-12 21:37 +0000
          Re: Storing a big amount of path names Ben Finney <ben+python@benfinney.id.au> - 2016-02-13 08:49 +1100
          Re: Storing a big amount of path names Matt Wheeler <funkyhat@gmail.com> - 2016-02-12 23:31 +0000
            Re: Storing a big amount of path names mkondrashin@gmail.com - 2016-02-13 12:19 -0800
  Re: Storing a big amount of path names srinivas devaki <mr.eightnoteight@gmail.com> - 2016-02-12 11:46 +0530

csiph-web