Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #79035 > unrolled thread

Re: Caching: Access a local file, but ensure it is up-to-date from a remote URL

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Caching: Access a local file, but ensure it is up-to-date from a remote URL Chris Angelico <rosuav@gmail.com> - 2014-10-13 18:51 +1100

#79035 — Re: Caching: Access a local file, but ensure it is up-to-date from a remote URL

On Mon, Oct 13, 2014 at 5:36 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
> So this is something similar to an HTTP object cache. Except where those
> are usually URL-focussed with the local files a hidden implementation
> detail, I want an API that focusses on the local files, with the remote
> requests a hidden implementation detail.

Potential issue: You may need some metadata storage as well as the
actual files. Or can you just ignore the Varies header etc etc etc,
and pretend that this URL represents a single blob of data no matter
what? I'm also dubious about relying on FS timestamps for critical
data, as it's very easy to bump the timestamp to current, which would
make your program think that the contents are fresh; but if that's
truly the only metadata needed, that might be safe to accept.

One way you could possibly do this is to pick up a URL-based cache
(even something stand-alone like Squid), and then create symlinks from
your canonically-named local files to the implementation-detail
storage space for the cache. Then you probe the URL and return its
contents. That guarantees that you're playing nicely with the rules of
HTTP (particularly if you have chained proxies, proxy authentication,
etc, etc - if you're deploying this to arbitrary locations, that might
be an issue), but at the expense of complexity.

ChrisA

[toc] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web