Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.acorn.apps > #2803

Re: Fresh packaging ideas

Cross-posted to 2 groups.

Show all headers | View raw

The following bytes were arranged on 19 Jan 2012 by Theo Markettos :

> In comp.sys.acorn.apps Martin Bazley <martin.bazley@blueyonder.co.uk> wrote:
> > If the main problem with the currently existing RiscPkg front-ends is
> > the lack of a flexible disc layout, then the main problem with the
> > libpkg back-end, and the reason I don't think the present system is
> > salvagable, is its reliance on repositories.
>
> I think you misunderstand what a repository is.  It supplies two things, a
> Packages file (a list of the packages available) and some package archives.

I know perfectly well what a repository is, thank you.

> Actually, the Packages file already contains URLs to each package archive.
> So a putative repository replacement merely has to fabricate a suitable
> Packages file, where the URLs can go off all over the place.  The Packages
> file is a necessity because it holds the dependency and description
> information but it, or the data structures that represent it, could easily
> be built from another source by libpkg.

The Packages file (or the Index file, as RiscPkg calls them) contains
nothing of the sort.  Both dependency and description information are
individually stored in each zip, in a file called RiscPkg.Control, and
all the Index file does is copy all of this information into a long list
in one file.  Believe me - I've built them.

I don't think you understood my proposal for URLs, either.  I'm not
suggesting a different way of associating URLs with package names - I'm
suggesting replacing package names *with* the URLs entirely.  They're
guaranteed to be unique, after all.

PackMan and the RiscPkg front-end already ship with a default list of
available packages, in the form of a link to the RiscPkg repo.  Why
couldn't such a thing be moved offline and distributed with each
archive?

> > My objections to repos are not ideological at all; they're purely
> > practical.  I think it would be wonderful if all RISC OS software could
> > be gathered together in one place, but I'm also sensible enough to
> > recognise that as head-in-the-clouds idealism, and that, for a variety
> > of reasons, it will never happen.
>
> The 'distributed repo' idea has some merits... as long as the risks can be
> kept under control too.

The main risk is that of software moving locations or being abandoned
and taken up by someone else, which can be viewed as very much as the
inverse of the notorious problem of managed data moving (or even being
copied) on a hard disc - i.e. easily surmountable.

> > My extremely sketchy plan for its replacement dispenses with repos
> > altogether, and instead of referring to each package by its name,
> > uniquely identifies each one by the URL its information should be
> > downloaded from.  Usually, this would be hosted on an author's own page,
> > hence creating the minimum of friction with the present system, although
> > I could foresee like-minded authors pooling webspace/hosting costs to
> > create a pseudo-'repository' of sorts - but the important thing is that
> > this wouldn't be compulsory, merely neater.  This enables much more
> > efficient dispersal of new software releases, and cuts out the middlemen
> > who guard the repos.  (Naturally, as at present, a rollback feature
> > should be available within the manager itself.)
>
> The question becomes, how do you get a full picture of what's available?
> ProgX depends on libFoo and BarMod.  I know the location of ProgX, but how
> do I find libFoo and BarMod?  Bearing in mind that libFoo might have
> recently moved servers, so that only the old version is at the site
> the ProgX author knew about 5 years ago when they wrote their program.

That was precisely the sort of circumstance I was thinking about when I
wrote the paragraph above.  I went into more detail later (so keep
reading).

> > What I'd really love would be for the absolute minimum specification of
> > a valid package to match up perfectly with the currently accepted
> > practice of distributing manually-installed software - that is to say, a
> > zip file containing an application, maybe a couple of preliminary text
> > files, possibly some extras/helper apps/resource files, and a skeleton
> > !Boot structure containing anything which should go in Resources,
> > !System, Choices, !Fonts, etc.  (That's assuming the presence of a
> > universal command-line boot merge utility, something currently lacking.)
> > Then everyone could transition to a package manager with the minimum of
> > effort, albeit obviously with all the bits which make package management
> > worthwhile missing (and 'malformed' zip files in the strange zoo of RISC
> > OS software could be problematic).
>
> Yes, that was one of my suggestions... a normal Zip but with some additional
> instructions to the package manager as to what should go where.

Exactly - but *without* the additional instructions, so any given zip
currently available (or at least most of them) could theoretically be
used in this manner.  It should be default behaviour for the manager to
scan the archive for directories called !Boot, !System or !Fonts anyway,
so it wouldn't be entirely dumb.  Of course, additional instructions
should preferably be made available, in order to gain the maximum
benefit - but we've got to start somewhere, otherwise we'll never get
started.

> > I realise some probably think I'm a heretic for not forcing people to do
> > things the 'proper' way, with dependency resolution et al., but I prefer
> > to think of myself as a realist, and at least this way you're training
> > people to use the manager.  Hopefully, much as Aemulor eased the
> > transition between the 26-bit and 32-bit eras, having something which at
> > least works, even if it's a bit of a bodge, will ultimately be more
> > productive than asking everybody to take the plunge simultaneously.
>
> Without dependency resolution or update checking you're essentially
> suggesting a simple Windows-style install wizard.  Which is fine, but it's
> not that useful on RISC OS (where drag and drop installs are simple anyway,
> give or take SysMerge etc).

Precisely - not that useful, but it's something to start from.  There's
a criminal lack of understanding in this thread of the fact that we
*need* somewhere to start from, no matter how primitive it may be at
first - and 'starting from' the current libpkg codebase will just
maintain the current woeful situation (most software unavailable, the
rest packaged by third parties with all the usual problems with that) in
perpetuity.

> > Above the absolute minimum baseline (which would obviously only be used
> > if absolutely necessary), the question becomes one of where to put
> > metadata - and how to get it, in the absence of a central index file,
> > into the manager.  It could ship with a long list of recommended
> > locations, but that couldn't cover everything.  While it'd be much
> > easier to get developers on board if they have to vary their current
> > building regime as little as possible, I think some sort of short
> > metadata file accompanying each zip is sadly going to be necessary, for
> > reasons of bandwidth apart from anything else.
>
> Such could be generated from the 'packaging wizard' - PackIt or similar.

It would probably end up that way, yes.

> > In the 'compatibility mode' case, the manager could fake this behaviour
> > by storing the filename of the zip, and the date stamp of the file on
> > the server at the point of the last fetch.  It could then decide whether
> > or not to download it again by comparing the stored date against the
> > current one.  (I don't know how technically feasible this is, or if
> > there is any kind of automated server-side distribution software which
> > causes the date stamp to be refreshed every 24 hours or so.)
>
> I think I'm failing to understand - won't that mean the file always comes up
> as 'new', even if it isn't?

Only if, for whatever reason, the file on the server continually had its
datestamp changed.  I'm hoping that doesn't happen in practice.
Certainly, if you were to install any of my software that way, it would
definitely work.

> Date stamps are a bit vulnerable at the server end: if you have to restore
> from backup and end up mangling the dates for some reason, you'd force
> everyone 'out there' to upgrade for no reason.

Yep.  I didn't say it was perfect, and a metadata file would guard
against this.

> > In the normal case, the metadata file would need to cope with problems
> > which don't arise in a centralised repo.  One of these is that the
> > file's location is imminently going to change, or has just changed.  In
> > this case, if the installed version is still up to date, the manager
> > updates the relevant entry in its package database but otherwise does
> > nothing.  Consideration also needs to be given to outdated dependency
> > files elsewhere on the net - I think the best solution would be for both
> > the distribution and the local manager to keep a record of all obsolete
> > URLs (of which there should never be very many, since sites, in general,
> > stay put), and for the manager include them in any database lookups.
>
> But who is managing this list of outdated dependency URLs?

Nobody.

> By definition, the person who created the old URL is unlikely to
> deprecate it.

Unless they planned to move their website.

> So someone else has to declare the URL outdated.

That'll be the person who wrote the superseding version in the first
place, then.

> Do you have 'dependency police', or can someone declare that someone
> else's work is outdated?  How do you manage that?

Like this:

PackageURL: riscpkg://example.com/libfoo/libfoo.txt
ObsoleteURLs: riscpkg://example.com/libfoo/libfoo_patch.txt,
riscpkg://other.website.com/software/libfoo/libfoo_ancient.txt

The zip file of the package itself would be the same as the metadata
file itself (so the PackageURL field would actually be the URL of the
exact same file it was read from), but ending in ".zip".  If any of
these URLs (or any URLs, really) ended in ".zip", the manager would
first check for an equivalent ".txt" (just in case the file was updated
to package format) and use that instead if found, otherwise fall back to
the 'install wizard' solution.

Naturally, the manager would maintain an offline database of all known
URLs, *including* the obsolete ones - precisely so it could check any
requested dependencies against them.  Even if you're installing libfoo
for the very first time, you'd still get all of its obsolete locations
as well.

The tricky bit is announcing the new home of libfoo.  In many ways, the
situation is very similar today (if people don't see the announcement
and keep checking the previous home page, they'll never have the new
one, and there's not much that can be done about that), but ideally, the
new maintainer would post on such as csa.announce the new URL, which
would point to the metadata, and when you clicked on it, the manager
would spot an entry in its "ObsoleteURLs" field which matched the name
of a package you already had installed, and update it for you.

In the planned move scenario, the libfoo_patch file would contain
exactly the same two fields, which would also be interpreted as a prompt
to update the local database, since the "PackageURL" didn't match the
URL of the file it was just read from.

That leaves as the remaining vulnerability users who install software
which lists an obsolete location of a dependency they don't have.  If
it's still there, then it'll be installed with no questions asked, which
poses the only real hazard.  Fortunately, it should also be fairly rare
(or at least no more common than it is now, which goes with my mantra
that fixing old problems isn't worth creating new ones).

> > Probably the biggest problem is how to find software in the first place
> > - which is where a lot of my original objections to repos came from.
> > One idea I had would be to dedicate an entire new URI scheme (say
> > riscpkg://) to the purpose, so that any clicks on it can be easily
> > claimed (via AcornURI) by the manager, and interpreted as pointing
> > towards (depending on filename) either a metadata file, or a
> > compatibility zip file.  This would result in the file being downloaded
> > and its contents (or filename/datestamp) being added to the local
> > database, after which a dialogue box would pop up asking you if you
> > wanted to install the software now.  To save clicks, website maintainers
> > could offer a limited version of a central index, accessed and
> > downloaded in the same way as a metadata file, but when the manager
> > opens it, it finds nothing but a list of URLs of other metadata
> > files/zips/indexes, each of which gets interpreted in turn.  (I'm not
> > sure making this scanning process re-entrant is really worthwhile, but
> > it'd be cool.)
>
> Android has the market:// URL format in a similar way... it works, but it's
> something of a pain if you're browsing on a non-Android device since it's
> used not only to install an app, but merely to find out about it (and some
> Android devices can't use Market at all).  After several years they got
> around to making a website with the same information.
>
> But having it merely as an install mechanism, and with a parallel http://
> way to download the archive for manual inspection, might be OK.

That would be included pretty much by definition.  The idea is that a
click on a riscpkg:// URL would add the data contained by the metadata
file or (if applicable) datestamp on the other end to the local
database, and then ask you if you want to install the file.  Answering
no would keep the location on the database for possible future
installation.

Interpreting a file full of such URLs would shortcut all this (and not
bring up the dialogue box for each one, since it would be redundant and
a nuisance), and be the standard way of populating an empty database.
Shipping one of these with the manager itself would do nicely, or
possibly have the manager come with only one entry preinstalled, which
is a list of default package URLs hosted on its own home page (with
priority set to 'required').

Updating this list would also be a neat fallback solution for the
obsolete dependency problem, forcing everybody to update the offending
entry in their local databases the moment they launch the manager.  Of
course, this is rather like a repo, but with the rather important
difference that it isn't compulsory for software to be in it.  As I
believe I said in my last post, repos are neater and have many
advantages, but most of these evaporate the moment you outlaw software
outside them.

> A web of links as you suggest sounds OK, though would have to think about
> the corner cases.

> What would be your initial conditions?  You download the package manager
> on a blank machine.  How does it find out about software you might want to
> install?

Well, as my proposals are mostly about finding sensible defaults, it
follows that the package manager wouldn't be distributed with an
absolutely blank database file, but one already filled up with as many
entries as possible, covering all common software and dependencies, and
hopefully most uncommon ones as well.  (Or with a list/link to list
suitable for automatic interpreting; see above.)

Getting more would simply be a matter of using Google (as opposed to the
much more unfriendly package search services which Linux has) to find
the software's home page, clicking on the presented riscpkg:// URL, and
selecting "Install now" on the resulting dialogue box.

-- 
  __<^>__
 / _   _ \         You always find something in the last place you look.
( ( |_| ) )
 \_>   <_/  ======================= Martin Bazley ==========================

Back to comp.sys.acorn.apps | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-15 17:02 +0000
  Re: Fresh packaging ideas Jim Nagel <jimnewsm10d@abbeypress.co.uk> - 2012-01-16 11:39 +0000
  Re: Fresh packaging ideas wpb <w.blatchley@yahoo.com> - 2012-01-16 05:49 -0800
    Re: Fresh packaging ideas Steve Fryatt <news@stevefryatt.org.uk> - 2012-01-16 19:01 +0000
    Re: Fresh packaging ideas Chris Evans <chris@cjemicros.co.uk> - 2012-01-17 13:04 +0000
      Re: Fresh packaging ideas wpb <w.blatchley@yahoo.com> - 2012-01-17 23:01 -0800
        Re: Fresh packaging ideas Chris Evans <chris@cjemicros.co.uk> - 2012-01-18 15:16 +0000
    Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-17 22:32 +0000
      Re: Fresh packaging ideas Matthew Phillips <spam2011m@yahoo.co.uk> - 2012-01-18 07:37 +0000
  Re: Fresh packaging ideas News Poster <workstuff@mail.com> - 2012-01-16 11:46 -0800
  Re: Fresh packaging ideas Alan <alan_baa@hotmail.com> - 2012-01-16 14:39 -0800
    Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-17 22:23 +0000
      Re: Fresh packaging ideas Alan <alan_baa@hotmail.com> - 2012-01-18 05:04 -0800
        Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-18 21:33 +0000
          Re: Fresh packaging ideas "David Holden" <SpamBin@apdl.co.uk> - 2012-01-19 08:03 +0000
            Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-19 11:46 +0000
            Re: Fresh packaging ideas Alan <alan_baa@hotmail.com> - 2012-01-19 05:45 -0800
              Re: Fresh packaging ideas News poster <news@mistymornings.net> - 2012-01-19 19:35 +0100
          Re: Fresh packaging ideas Matthew Phillips <spam2011m@yahoo.co.uk> - 2012-01-19 08:09 +0000
            Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-19 19:36 +0000
              Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-19 20:27 +0000
                Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-19 21:18 +0000
                Re: Fresh packaging ideas Matthew Phillips <spam2011m@yahoo.co.uk> - 2012-01-19 23:01 +0000
                Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-20 20:22 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-20 09:02 +0100
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-20 09:04 +0100
                Re: Fresh packaging ideas Jess Hampshire <jesshampshire@googlemail.com> - 2012-01-23 19:43 +0000
                Re: Fresh packaging ideas wpb <w.blatchley@yahoo.com> - 2012-01-23 21:30 -0800
                Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-25 16:18 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-25 18:15 +0100
              Re: Fresh packaging ideas Matthew Phillips <spam2011m@yahoo.co.uk> - 2012-01-19 21:55 +0000
                Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-19 23:02 +0000
                Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-20 19:48 +0000
                Re: Fresh packaging ideas Matthew Phillips <spam2011m@yahoo.co.uk> - 2012-01-20 20:44 +0000
                Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-20 21:10 +0000
                Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-21 00:51 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-21 04:39 +0100
                Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-21 15:34 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-21 04:32 +0100
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-21 04:44 +0100
          Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-19 13:53 +0000
            Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-19 20:56 +0000
              Re: Fresh packaging ideas Chris Evans <chris@cjemicros.co.uk> - 2012-01-20 13:32 +0000
                Re: Fresh packaging ideas Patric@invalid.com - 2012-01-20 16:16 +0100
                Re: Fresh packaging ideas druck <news@druck.org.uk> - 2012-01-21 09:55 +0000
                Re: Fresh packaging ideas Fred Bambrough <fred@[127.0.0.1]> - 2012-01-21 11:00 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-21 19:04 +0100
                Re: Fresh packaging ideas Dave Symes <dave@triffid.co.uk> - 2012-01-21 19:09 +0000
                Re: Fresh packaging ideas Ron Briscoe <ron.briscoe@blueyonder.co.uk> - 2012-01-21 20:18 +0000
                Re: Fresh packaging ideas Patric@invalid.com - 2012-01-21 23:21 +0100
                Re: Fresh packaging ideas Stuart <Spambin@argonet.co.uk> - 2012-01-21 23:28 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-21 04:18 +0100
                Re: Fresh packaging ideas Alan <alan_baa@hotmail.com> - 2012-01-22 07:51 -0800
                Re: Fresh packaging ideas Frank de Bruijn <zuiderduin@hotmail.com> - 2012-01-23 08:13 +0100
              Re: Fresh packaging ideas Alan <alan_baa@hotmail.com> - 2012-01-23 05:06 -0800
                Re: Fresh packaging ideas Martin Bazley <martin.bazley@blueyonder.co.uk> - 2012-01-23 21:32 +0000
                Re: Fresh packaging ideas Alan <alan_baa@hotmail.com> - 2012-01-24 05:05 -0800
                Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-25 22:41 +0000
                Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-26 15:33 +0100
  Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-17 06:39 +0100
    Re: Fresh packaging ideas druck <news@druck.org.uk> - 2012-01-17 23:10 +0000
      Re: Fresh packaging ideas Rick Murray <heyrickmail-usenet@yahoo.co.uk> - 2012-01-18 09:26 +0100
    Re: Fresh packaging ideas Theo Markettos <theom+news@chiark.greenend.org.uk> - 2012-01-17 23:10 +0000
  Re: Fresh packaging ideas Stuart <Spambin@argonet.co.uk> - 2012-01-17 15:20 +0000
    Re: Fresh packaging ideas Gavin Wraith <gavin@wra1th.plus.com> - 2012-01-17 16:33 +0000
      Re: Fresh packaging ideas Chris Johnson <chrisjohnson+news@spamcop.net> - 2012-01-17 17:02 +0000
        Re: Fresh packaging ideas Tim Hill <tim@invalid.org.uk> - 2012-01-21 15:12 +0000
  Re: Fresh packaging ideas  <news@iank.org.uk> - 2012-01-28 12:19 +0000

csiph-web