Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #33295

Re: Generate unique ID for URL

Path csiph.com!usenet.pasdenom.info!news.albasani.net!news2.arglkargh.de!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path <rosuav@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.002
X-Spam-Evidence '*H*': 1.00; '*S*': 0.00; '(at': 0.03; 'retrieval': 0.05; 'say,': 0.05; 'bytes)': 0.09; 'sub': 0.09; 'suggest': 0.11; 'disk.': 0.16; 'dwarfed': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'storing': 0.16; 'structure.': 0.16; 'subject:Generate': 0.16; 'subject:URL': 0.16; 'url.': 0.16; 'wed,': 0.16; 'wrote:': 0.17; 'received:209.85.214.174': 0.21; 'example': 0.23; 'header:In-Reply-To:1': 0.25; 'disk': 0.27; 'message-id:@mail.gmail.com': 0.27; 'app.': 0.29; 'arithmetic': 0.29; 'hash': 0.29; 'probably': 0.29; "i'm": 0.29; 'folder': 0.30; 'figure': 0.30; 'could': 0.32; 'limitations': 0.33; 'to:addr :python-list': 0.33; "can't": 0.34; 'received:google.com': 0.34; 'nov': 0.35; 'richard': 0.35; 'pm,': 0.35; 'received:209.85': 0.35; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'store': 0.38; 'files': 0.38; 'some': 0.38; 'to:addr:python.org': 0.39; 'received:209.85.214': 0.39; 'header:Received:5': 0.40; '100': 0.78; 'viewed': 0.78; 'locally': 0.84; 'average': 0.93
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=OX5P+rUstKVCEgCM1uZwfrnHKhXHmdjLzyYe1jkpG/8=; b=XmfSt3nE8FvgBJMRUaWSUY3N9u4m0JZvAve9Exz/qUxP9JvctQ4Qct29emoZJN9XPI tEQ6oDTurtgWxqhd87o/DcA9nMV0JwcX0UZI8r0cg6jiALbInYZm0F4ni6Nwk+oyxdxb u4K8tVztVfA7unCoeioh2hcwI5wOJLK/+1uiv5K5fE5imzTFIz+dWPaRaeiR9ZkJxmGz YtKgZ+lctWFOOn0oml2Me2YtQ5YxlleHpQUVwehpUx+yhaw0pyVGocdyyG+lVmZ1EhTn ZQCY9aMV+xc9ETST33Wq+C4VVH7f3+0tGmK1plYCJD6haEcxhFtCV/9r18fCTrNDLyKq m4jw==
MIME-Version 1.0
In-Reply-To <1ce88f36-bfc7-4a55-89f8-70d1645d27ad@googlegroups.com>
References <0692e6a2-343c-4eb0-be57-fe5c815efb99@googlegroups.com> <roy-862116.20390413112012@news.panix.com> <1ce88f36-bfc7-4a55-89f8-70d1645d27ad@googlegroups.com>
Date Wed, 14 Nov 2012 15:06:10 +1100
Subject Re: Generate unique ID for URL
From Chris Angelico <rosuav@gmail.com>
To python-list@python.org
Content-Type text/plain; charset=ISO-8859-1
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.3661.1352865978.27098.python-list@python.org> (permalink)
Lines 14
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1352865978 news.xs4all.nl 6869 [2001:888:2000:d::a6]:60731
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:33295

Show key headers only | View raw

On Wed, Nov 14, 2012 at 2:25 PM, Richard <richardbp@gmail.com> wrote:
> So the use case - I'm storing webpages on disk and want a quick retrieval system based on URL.
> I can't store the files in a single directory because of OS limitations so have been using a sub folder structure.
> For example to store data at URL "abc": a/b/c/index.html
> This data is also viewed locally through a web app.
>
> If you can suggest a better approach I would welcome it.

The cost of a crypto hash on the URL will be completely dwarfed by the
cost of storing/retrieving on disk. You could probably do some
arithmetic and figure out exactly how many URLs (at an average length
of, say, 100 bytes) you can hash in the time of one disk seek.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 15:20 -0800
  Re: Generate unique ID for URL John Gordon <gordon@panix.com> - 2012-11-13 23:34 +0000
    Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 15:56 -0800
      Re: Generate unique ID for URL Chris Kaynor <ckaynor@zindagigames.com> - 2012-11-13 16:26 -0800
      Re: Generate unique ID for URL Richard Baron Penman <richardbp@gmail.com> - 2012-11-14 11:41 +1100
        Re: Generate unique ID for URL Johannes Bauer <dfnsonfsduifb@gmx.de> - 2012-11-14 10:44 +0100
          Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-14 03:14 -0800
      Re: Generate unique ID for URL Christian Heimes <christian@python.org> - 2012-11-14 01:43 +0100
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 16:50 -0800
          Re: Generate unique ID for URL Christian Heimes <christian@python.org> - 2012-11-14 02:05 +0100
      Re: Generate unique ID for URL Christian Heimes <christian@python.org> - 2012-11-14 01:59 +0100
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 17:18 -0800
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 17:18 -0800
  Re: Generate unique ID for URL Miki Tebeka <miki.tebeka@gmail.com> - 2012-11-13 16:13 -0800
    Re: Generate unique ID for URL Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-11-14 02:04 +0000
      Re: Generate unique ID for URL Steve Howell <showell30@yahoo.com> - 2012-11-13 18:32 -0800
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 19:12 -0800
  Re: Generate unique ID for URL Roy Smith <roy@panix.com> - 2012-11-13 20:39 -0500
    Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 19:25 -0800
      Re: Generate unique ID for URL Roy Smith <roy@panix.com> - 2012-11-13 22:38 -0500
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 19:56 -0800
      Re: Generate unique ID for URL Chris Angelico <rosuav@gmail.com> - 2012-11-14 15:06 +1100
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 20:14 -0800
        Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 20:14 -0800
    Re: Generate unique ID for URL Richard <richardbp@gmail.com> - 2012-11-13 19:27 -0800
    Re: Generate unique ID for URL Johannes Bauer <dfnsonfsduifb@gmx.de> - 2012-11-14 12:29 +0100
      Re: Generate unique ID for URL Dave Angel <d@davea.name> - 2012-11-14 07:33 -0500
        Re: Generate unique ID for URL Johannes Bauer <dfnsonfsduifb@gmx.de> - 2012-11-14 14:00 +0100

csiph-web