Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #174517 > unrolled thread

Layers for the package manager

Back to article view | Back to linux.debian.user

Contents

  Layers for the package manager Nicolas George <george@nsup.org> - 2016-11-11 20:40 +0100
    Re: Layers for the package manager Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2016-11-11 23:30 +0100
      Re: Layers for the package manager Nicolas George <george@nsup.org> - 2016-11-12 21:20 +0100
        Re: Layers for the package manager Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2016-11-12 22:10 +0100
          Re: Layers for the package manager Nicolas George <george@nsup.org> - 2016-11-17 11:00 +0100
        Re: Layers for the package manager Jonathan Dowland <jmtd@debian.org> - 2016-11-14 18:50 +0100
    Re: Layers for the package manager Don Armstrong <don@debian.org> - 2016-11-15 18:30 +0100
      Re: Layers for the package manager Nicolas George <george@nsup.org> - 2016-11-17 10:50 +0100
        Re: Layers for the package manager Don Armstrong <don@debian.org> - 2016-11-17 16:30 +0100

#174517 — Layers for the package manager

[Multipart message — attachments visible in raw view] — view raw

There is a feature in Debian that I find missing in Debian, I wonder if
other people would agree with me.

I call that feature "layered package manager".

Here is what it means. The system has "layers" defined, organized as a
directed acyclic graph, and packages are contained in layers. For example:

         base
       /   |  \
      /    |   \
     v     v    v
nonfree  gnome  texlive
     \    |      |
      \   |      |
       v  v      |
     proprep     |
            \    |
             \   |
              v  v
               all

The idea is this: packages in a layer "see" the packages in the same layers
and in lower layers but not the packages in higher layers (in the drawing,
low and high are reversed). By "see", I mean the dependencies: a broken
dependencies in the proprep layer does not prevent me from upgrading
something in the gnome layer, or a broken dependency in the nonfree layer
does not block an upgrade in base.

When working in a layer, everything works exactly as if the higher layers
did not exist.

The lower layers are visible, though, and merged. For example, in the above
example, when installing something in proprep, if it needs a dependency that
is installed in either nonfree, base or gnome, it will just use it. If is is
not installer anywhere, then it gets installed in proprep itself.

Of course, it means that packages in higher layers can easily become broken.
It is not a bug, it is a feature. Better a system with a few inessential
packages partially broken than a system vulnerable because a security
upgrade on a base package could not be performed. Or than removing the
packages altogether.

A higher-level tool would be necessary to help handling things. For example,
if I try to remove a gnome package necessary for proprep, it proposes me to
reinstall it in proprep instead. Or it would propose a command to perform a
full upgrade in each layer in turn, in a single step.

Users need to be able to choose in which layer they work, including changing
on the fly. The admin should be able to choose the layer for all daemons
too.

Of course, a lot of details would need to be ironed. What happens if the
same package comes in different version from several parents? Can we remove
(=hide) in a higher layer a package from a lower one? Etc.

But I think it would make a very useful tool. Here are the benefits I
expect:

Being able to install incompatible packages at the same time. Of course, I
would need to switch layer to use them both, but that is not a big deal.

Isolate servers. Run servers in a layer where the compilers and other
development tools are not visible. It does not actually add any security but
it may help mitigate or slow down an attack.

Install packages on different filesystems depending on their usefulness:
base system on small SSD, desktop environment and other large packages on
larger but slower HDD, huge docs and games on even slower NAS.

Most important: Being able to add third-party repositories with less worry.

Right now, if I consider adding a third-party repository, or installing a
third-party package, I need to trust both its honesty (of course, any
package can contain a backdoor and take root access) and it seriousness.
With layers, I still need to trust its honesty, of course, but if I install
it in a separate layer, I do not have to worry that it may break something
else or block a security upgrade. It isolated.

So, this is what I dream of. Does anybody have remarks? Know projects with
similar goals? Distributions that have that feature? Other means of
achieving a similar result with Debian?

Regards,

-- 
  Nicolas George

[toc] | [next] | [standalone]

#174549

On 11-11-2016 17:34, Nicolas George wrote:
> Being able to install incompatible packages at the same time.
>
> Isolate servers. Run servers in a layer where the compilers and other
> development tools are not visible. It does not actually add any security but
> it may help mitigate or slow down an attack.
>
> Most important: Being able to add third-party repositories with less worry.
>
> So, this is what I dream of. Does anybody have remarks? Know projects with
> similar goals? Distributions that have that feature? Other means of
> achieving a similar result with Debian?

docker does not work as you describe (and I know of nothing that does
anything close to your idea), but it does suit some of the uses you
mentioned (the ones quoted).


-- 
Preguiça é o habito de descansar antes da fadiga.

Eduardo M KALINOWSKI
eduardo@kalinowski.com.br

[toc] | [prev] | [next] | [standalone]

#174608

[Multipart message — attachments visible in raw view] — view raw

Le primidi 21 brumaire, an CCXXV, Eduardo M KALINOWSKI a écrit :
> docker does not work as you describe (and I know of nothing that does
> anything close to your idea), but it does suit some of the uses you
> mentioned (the ones quoted).

If I understand correctly how Docker works, its images are big blobs
that contain the program they are meant to distribute plus all its
dependencies. Am I mistaken?

If it works like that, that means when the next OpenSSL security issue
is found, we have to cross our fingers very tightly and hope whoever
released the image will release an update with a fixed library. With
what I have in mind, unless the maintainer of the third-party repository
did something very wrong, its packages will be dynamically linked with
OpenSSL from the base system, and benefit from the updates immediately.

It makes a big difference: in one case, you have to trust the third
party to do a good job and continue that way in the future, on the other
case you only have to trust it to do a not-bad job once.

Personally, I would rather unpack a dynamically-linked binary somewhere
in /opt and install the dependencies myself than use a package system
with bundled libraries. Or, of course, install from source.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#174610

On 12-11-2016 18:18, Nicolas George wrote:
> If I understand correctly how Docker works, its images are big blobs
> that contain the program they are meant to distribute plus all its
> dependencies. Am I mistaken?
>
> If it works like that, that means when the next OpenSSL security issue
> is found, we have to cross our fingers very tightly and hope whoever
> released the image will release an update with a fixed library. With
> what I have in mind, unless the maintainer of the third-party repository
> did something very wrong, its packages will be dynamically linked with
> OpenSSL from the base system, and benefit from the updates immediately.
>
> It makes a big difference: in one case, you have to trust the third
> party to do a good job and continue that way in the future, on the other
> case you only have to trust it to do a not-bad job once.
>
> Personally, I would rather unpack a dynamically-linked binary somewhere
> in /opt and install the dependencies myself than use a package system
> with bundled libraries. Or, of course, install from source.
>

The end result is indeed, for practical purposes, a big package with
everything needed to run an application. But I don't expect you to
simply download pre-built blobs, but rather to build your own.

It's easier to understand from an example. Take a look at this
Dockerfile:
https://gist.github.com/rmoehn/1d82f433f517e3002124df52f7a73678 .
Basically it says "start with a minimal debian stable system, install
some packages, then make some necessary configurations". Docker
downloads the base image (the minimal debian system), and executes the
rest of the commands to create a new image that includes the installed
packages, resulting in a self-contained image that allows execution of
the application in question.

[That's a huge simplification, but the end result is roughly as
described. We need not concern with the internals.]

[If you don't want to trust the creator of the base image, you can
create your own.]

I've used that Dockerfile to run anki (which is currently broken in
testing). It's kind of a chroot, but with more isolation. Also, if I
wanted to run another program from stable, I could build another docker
image for that, but due to the clever way docker works, there'd be only
one copy of the base system plus two layers, one for each image, with
only the applications in question. That's a big advantage in relation to
chroots or virtual machines.

And yet, even if a base layer is shared, docker images are completely
isolated from one another and from the host system. If you want to share
data, you need to explicitly configure that.

That does solve the isolation problem, and allows you to run packages
from different repositories simultaneously, with different versions of
libraries if necessary; and allows you to install packages from
untrusted sources (or that are not available as .deb's) without messing
with your "real" system.

It does not solve the problem you mention: if there is an update of
OpenSSL, the images will continue to use the old version unless you
rebuild them. The process can be automated, but at least you'll need to
run a command to rebuild the images, and this can be time consuming.

Now might be a good time to dive into some of the internals, such as how
the images and layers work:
https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/
That might give you some ideas for your solution. Take a look also at
the pages "AUFS storage driver in practice" and "OverlayFS storage in
practice". While you won't be able to do what you want with docker,
perhaps you can get some ideas. I'd guess you'd need some kind of
layering like done by docker, but sometimes changing the bottom layers
(which is not possible with docker - only the topmost layer is ever
changed).


-- 
Eduardo M KALINOWSKI
eduardo@kalinowski.com.br

[toc] | [prev] | [next] | [standalone]

#174811

[Multipart message — attachments visible in raw view] — view raw

Le duodi 22 brumaire, an CCXXV, Eduardo M KALINOWSKI a écrit :
>							    Also, if I
> wanted to run another program from stable, I could build another docker
> image for that, but due to the clever way docker works, there'd be only
> one copy of the base system plus two layers, one for each image, with
> only the applications in question. That's a big advantage in relation to
> chroots or virtual machines.

I did not know Docker worked that way. It is interesting, and better
than I thought.

> And yet, even if a base layer is shared, docker images are completely
> isolated from one another and from the host system. If you want to share
> data, you need to explicitly configure that.
> 
> That does solve the isolation problem, and allows you to run packages
> from different repositories simultaneously, with different versions of
> libraries if necessary; and allows you to install packages from
> untrusted sources (or that are not available as .deb's) without messing
> with your "real" system.
> 
> It does not solve the problem you mention: if there is an update of
> OpenSSL, the images will continue to use the old version unless you
> rebuild them. The process can be automated, but at least you'll need to
> run a command to rebuild the images, and this can be time consuming.

Indeed. Compared to what I have in mind, it seems it would work for the
leaves/sinks of the graph system, but not the inner nodes.

Also, the isolation is kind of a mis-feature in this particular case

> Now might be a good time to dive into some of the internals, such as how
> the images and layers work:
> https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/
> That might give you some ideas for your solution. Take a look also at
> the pages "AUFS storage driver in practice" and "OverlayFS storage in
> practice". While you won't be able to do what you want with docker,
> perhaps you can get some ideas. I'd guess you'd need some kind of
> layering like done by docker, but sometimes changing the bottom layers
> (which is not possible with docker - only the topmost layer is ever
> changed).

Thanks for the pointers. I definitely do not have time to actually work
on it as a project, but knowing a few pointers will do no harm.
Union-capable filesystems were indeed something that I thought could
help an hypothetic implementation.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#174707

[Multipart message — attachments visible in raw view] — view raw

On Sat, Nov 12, 2016 at 09:18:26PM +0100, Nicolas George wrote:
> If I understand correctly how Docker works, its images are big blobs
> that contain the program they are meant to distribute plus all its
> dependencies. Am I mistaken?

That's broadly true, at least, they are presented to the user that way.

> If it works like that, that means when the next OpenSSL security issue
> is found, we have to cross our fingers very tightly and hope whoever
> released the image will release an update with a fixed library. 

If the dockerized application you were using did not use OpenSSL, you don't
need to do anything.

If it does, then it needs to be rebuilt.

As a user, you could wait for the image author to rebuild it, then pull the
newer version. It's quite likely the image author based their image on top of
another, and that the parent image (or a parent image somewhere up the chain)
is the "owner" of the problem. For example, if the image you were using had
been built on top of the "debian" image, the image author would have to wait
until the author/owner of the "debian" image updated, first. (Note: that isn't
the Debian project, confusingly!)

This is pretty bad.

Another option is, you create a new derivative image yourself, which is
built on top of the application image you were using, and updates OpenSSL.
That would be roughly as simple as a Dockerfile with two instructions

	FROM some-application-image
	RUN apt-get update && apt-get upgrade -y

(the above fed to "docker build -t my-new-image-name <location to Dockerfile>")

The abstraction is already a bit broken though, because you need to know
that the image was based on a Debian-like OS; that OS needs to have updated
the OpenSSL library; the application has to continue to function with the
new library version (and that might not be tested yet)

And none of the above is automatic.

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#174763

On Fri, 11 Nov 2016, Nicolas George wrote:
> There is a feature in Debian that I find missing in Debian, I wonder if
> other people would agree with me.
> 
> I call that feature "layered package manager".

This sounds like you're looking for debtags, which gives you the ability
to subset packages by different tags. (For example, if you wanted all of
the package management tools:

debtags search admin::package-management;

gives you all of the packages which are involved in package management.)


-- 
Don Armstrong                      https://www.donarmstrong.com

"She decided what she wished to happen and then assumed that reality
would bend to her wishes." [...] "Reality doesn't indulge wishes."
 -- Terry Goodkind _Phantom_ p133

[toc] | [prev] | [next] | [standalone]

#174810

[Multipart message — attachments visible in raw view] — view raw

Le quintidi 25 brumaire, an CCXXV, Don Armstrong a écrit :
> This sounds like you're looking for debtags, which gives you the ability
> to subset packages by different tags. (For example, if you wanted all of
> the package management tools:
> 
> debtags search admin::package-management;
> 
> gives you all of the packages which are involved in package management.)

Uh? Can you explain me how it can be used to install incompatible
packages or packages from semi-trusted repositories?

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#174817

On Thu, 17 Nov 2016, Nicolas George wrote:
> Le quintidi 25 brumaire, an CCXXV, Don Armstrong a écrit :
> > This sounds like you're looking for debtags, which gives you the ability
> > to subset packages by different tags. (For example, if you wanted all of
> > the package management tools:
> > 
> > debtags search admin::package-management;
> > 
> > gives you all of the packages which are involved in package management.)
> 
> Uh? Can you explain me how it can be used to install incompatible
> packages or packages from semi-trusted repositories?

Sorry; I misunderstood what you were trying to do.

debtags will help you define which packages go in what layers, but they
won't do anything to help you with the rest of the features that you're
describing.


-- 
Don Armstrong                      https://www.donarmstrong.com

Good people are good because they've come to wisdom through failure.
We get very little wisdom from success, you know.
 -- William Saroyan _My Heart's in the Highlands_

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web