Re: Caching: Access a local file, but ensure it is up-to-date from a remote URL

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!eternal-september.org!feeder.eternal-september.org!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.001
X-Spam-Evidence	'H': 1.00; 'S': 0.00; '(even': 0.05; 'represents': 0.05; 'cache': 0.07; 'needed,': 0.07; 'nicely': 0.07; 'subject:file': 0.07; 'expense': 0.09; 'http': 0.09; 'issue:': 0.09; 'pretend': 0.09; 'api': 0.11; 'cc:addr:python- list': 0.11; 'bump': 0.16; 'chained': 0.16; 'finney': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'issue),': 0.16; 'proxies,': 0.16; 'subject: \n ': 0.16; 'subject:URL': 0.16; 'timestamp': 0.16; 'timestamps': 0.16; 'varies': 0.16; 'files.': 0.16; 'ignore': 0.16; 'wrote:': 0.18; 'rules': 0.22; 'cc:addr:python.org': 0.22; 'proxy': 0.24; 'header': 0.24; 'mon,': 0.24; 'cc:2**0': 0.24; 'possibly': 0.26; 'header:In-Reply-To:1': 0.27; 'message-id:@mail.gmail.com': 0.30; "i'm": 0.30; 'requests': 0.31; 'usually': 0.31; '13,': 0.31; 'critical': 0.32; 'actual': 0.34; 'subject:from': 0.34; 'could': 0.34; 'except': 0.35; 'something': 0.35; 'etc': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'data,': 0.36; 'similar': 0.36; 'remote': 0.38; 'ben': 0.38; 'files': 0.38; 'pm,': 0.38; 'space': 0.40; 'easy': 0.60; 'truly': 0.60; 'matter': 0.61; "you're": 0.61; 'pick': 0.64; 'detail.': 0.68; 'safe': 0.72; '5:36': 0.84; 'blob': 0.84; 'etc,': 0.84; 'contents.': 0.91; 'probe': 0.91; 'to:none': 0.92
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=avD6N5XaRtDXInbN940FVBLRkK+VdtBUPllGS/5SwjI=; b=Z/DgDeIp4iwBaTVuMvJN9tuGJkQHE1ZeFofez5nPl48thtgRHTpjSawzmnD4FtkCMm pDLk/zEkTKlBuO7wccjh/rzYH352/F/jKkPksSQk/mLCFsyQlHOGi83GjJLhEfWAkTjq 1FFdtJrKpbYq4B2kKCENoIjm+B75Kq3c6LqnjFo6ZU+j9cNo2N8Ti1FrX2E8dRYpLvjN kXoiyRn9rb+LuG6CfNs/b4h0GNufbgUvIF7xwgqw4itiOimaVgkpbIuAbNFfPvuX3LcD v/7+ZWm3yJ28fk/Q09WxQ2Bqx+sU2OtfyHaF3CckQajmfkR3DuY4Wo9+3z5j7GxsftCg Uwqg==
MIME-Version	1.0
X-Received	by 10.50.117.65 with SMTP id kc1mr28410594igb.34.1413186684381; Mon, 13 Oct 2014 00:51:24 -0700 (PDT)
In-Reply-To	<85r3ycxyps.fsf@benfinney.id.au>
References	<85r3ycxyps.fsf@benfinney.id.au>
Date	Mon, 13 Oct 2014 18:51:24 +1100
Subject	Re: Caching: Access a local file, but ensure it is up-to-date from a remote URL
From	Chris Angelico <rosuav@gmail.com>
Cc	"python-list@python.org" <python-list@python.org>
Content-Type	text/plain; charset=UTF-8
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.14813.1413186688.18130.python-list@python.org> (permalink)
Lines	24
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1413186688 news.xs4all.nl 2845 [2001:888:2000:d::a6]:42043
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:79035

Show key headers only | View raw

On Mon, Oct 13, 2014 at 5:36 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
> So this is something similar to an HTTP object cache. Except where those
> are usually URL-focussed with the local files a hidden implementation
> detail, I want an API that focusses on the local files, with the remote
> requests a hidden implementation detail.

Potential issue: You may need some metadata storage as well as the
actual files. Or can you just ignore the Varies header etc etc etc,
and pretend that this URL represents a single blob of data no matter
what? I'm also dubious about relying on FS timestamps for critical
data, as it's very easy to bump the timestamp to current, which would
make your program think that the contents are fresh; but if that's
truly the only metadata needed, that might be safe to accept.

One way you could possibly do this is to pick up a URL-based cache
(even something stand-alone like Squid), and then create symlinks from
your canonically-named local files to the implementation-detail
storage space for the cache. Then you probe the URL and return its
contents. That guarantees that you're playing nicely with the rules of
HTTP (particularly if you have chained proxies, proxy authentication,
etc, etc - if you're deploying this to arbitrary locations, that might
be an issue), but at the expense of complexity.

ChrisA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Caching: Access a local file, but ensure it is up-to-date from a remote URL Chris Angelico <rosuav@gmail.com> - 2014-10-13 18:51 +1100

csiph-web