Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #58755 > unrolled thread

Generating a cloud / VM kernel package

Back to article view | Back to linux.debian.kernel

Contents

  Generating a cloud / VM kernel package Thomas Goirand <zigo@debian.org> - 2017-08-26 12:00 +0200
    Re: Generating a cloud / VM kernel package Martin Zobel-Helas <zobel@debian.org> - 2017-08-26 12:40 +0200
      Re: Generating a cloud / VM kernel package Ben Hutchings <ben@decadent.org.uk> - 2017-08-26 18:20 +0200
        Re: Generating a cloud / VM kernel package Noah Meyerhans <noahm@debian.org> - 2017-08-28 01:50 +0200
          Re: Generating a cloud / VM kernel package Ben Hutchings <ben@decadent.org.uk> - 2017-08-28 02:40 +0200
            Re: Generating a cloud / VM kernel package Noah Meyerhans <noahm@debian.org> - 2017-08-28 08:00 +0200
      Re: Generating a cloud / VM kernel package Thomas Goirand <zigo@debian.org> - 2017-08-27 16:20 +0200
        Re: Generating a cloud / VM kernel package Noah Meyerhans <noahm@debian.org> - 2017-08-28 16:10 +0200
    Re: Generating a cloud / VM kernel package Ben Hutchings <ben@decadent.org.uk> - 2017-08-26 18:20 +0200
    Re: Generating a cloud / VM kernel package Bastian Blank <waldi@debian.org> - 2017-08-28 14:10 +0200

#58755 — Generating a cloud / VM kernel package

Dear Kernel maintainers,

As you may know, it's been years that Ubuntu is shipping a kernel
designed for the cloud. Such a kernel is simply a version of the kernel
that is stripped down for running on VMs. The point here is that VMs do
not need all the drivers that we typically build for the generic Debian
kernel (and if one still needs it, a fallback to the generic kernel is
always possible). This makes the kernel binary package a lot smaller,
and also potentially reduces the surface of attack in case of a security
problem. For example, we wouldn't need ax25, appletalk and such, which
are unfortunately automatically loaded in case matching packets are
received by the kernel, and which have been proven to be problematic in
terms of security maintenance. Most hardware drivers would also go away.

Since it is only a mater of *removing* some modules, I don't think
adding a cloud / VM kernel flavor would be a lot of maintenance. Though
of course, as I wouldn't be the one doing it, it is not up to me to
judge the amount of work.

Could we see this happening in Debian? Please let us know your thoughts.

Cheers,

Thomas Goirand (zigo)

[toc] | [next] | [standalone]

#58756

Hi, 

On Sat Aug 26, 2017 at 11:48:22 +0200, Thomas Goirand wrote:
> Dear Kernel maintainers,
> 
> As you may know, it's been years that Ubuntu is shipping a kernel
> designed for the cloud. Such a kernel is simply a version of the kernel
> that is stripped down for running on VMs. The point here is that VMs do
> not need all the drivers that we typically build for the generic Debian
> kernel (and if one still needs it, a fallback to the generic kernel is
> always possible). This makes the kernel binary package a lot smaller,
> and also potentially reduces the surface of attack in case of a security
> problem. For example, we wouldn't need ax25, appletalk and such, which
> are unfortunately automatically loaded in case matching packets are
> received by the kernel, and which have been proven to be problematic in
> terms of security maintenance. Most hardware drivers would also go away.
> 
> Since it is only a mater of *removing* some modules, I don't think
> adding a cloud / VM kernel flavor would be a lot of maintenance. Though
> of course, as I wouldn't be the one doing it, it is not up to me to
> judge the amount of work.
> 
> Could we see this happening in Debian? Please let us know your thoughts.

I personaly think this is a good idea in general. Most cloud providers
will probably want/love this, esp. when it comes to specifica of their
environments. On the other hand I have some concerns:

a) we need to decide then if we need one kernel flavour for each cloud
provider or if we can agree on a basic set of kernel compile options
that every cloud provider can use.

b) most kernels Debian ships are kernels that have most drivers needed
as modules, so even though the kernel images are big, the kernel should
only load modules it really needs.

Thomas, can you elaborate why you think this a good idea? Is this about
boot time of the kernel image? The thing I really do not want to have is
additional kernel source uploads to the archive for just those cloud
kernel images, but you already considered that a bad idea (from what I
read between your lines).

Cheers,
Martin

-- 
 Martin Zobel-Helas <zobel@debian.org>    Debian System Administrator
 Debian & GNU/Linux Developer                       Debian Listmaster
 http://about.me/zobel                               Debian Webmaster
 GPG Fingerprint:  6B18 5642 8E41 EC89 3D5D  BDBB 53B1 AC6D B11B 627B 

[toc] | [prev] | [next] | [standalone]

#58761

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2017-08-26 at 12:15 +0200, Martin Zobel-Helas wrote:
[...]
> Thomas, can you elaborate why you think this a good idea? Is this about
> boot time of the kernel image? The thing I really do not want to have is
> additional kernel source uploads to the archive for just those cloud
> kernel images, but you already considered that a bad idea (from what I
> read between your lines).

When the Google Cloud people talked to me about slow booting, it turned
out that reconfiguring initramfs-tools to MODULES=dep made a big
improvement.  That is likely to be a sensible configuration for most
cloud images.

Ben.

-- 
Ben Hutchings
One of the nice things about standards is that there are so many of
them.

[toc] | [prev] | [next] | [standalone]

#58798

[Multipart message — attachments visible in raw view] — view raw

On Sat, Aug 26, 2017 at 05:18:45PM +0100, Ben Hutchings wrote:
> > Thomas, can you elaborate why you think this a good idea? Is this about
> > boot time of the kernel image? The thing I really do not want to have is
> > additional kernel source uploads to the archive for just those cloud
> > kernel images, but you already considered that a bad idea (from what I
> > read between your lines).
> 
> When the Google Cloud people talked to me about slow booting, it turned
> out that reconfiguring initramfs-tools to MODULES=dep made a big
> improvement.  That is likely to be a sensible configuration for most
> cloud images.

I'm not sure that'll work for us. The image generation is not generally
expected to occur on cloud instances (though in practice it certainly
may).

OTOH, the list of required modules may be small enough for us to
enumerate the ones we need for booting in /etc/initramfs-tools/modules.
I will look into this, and we'll see what it does to boot times.

noah

[toc] | [prev] | [next] | [standalone]

#58799

[Multipart message — attachments visible in raw view] — view raw

On Sun, 2017-08-27 at 16:38 -0700, Noah Meyerhans wrote:
> On Sat, Aug 26, 2017 at 05:18:45PM +0100, Ben Hutchings wrote:
> > > Thomas, can you elaborate why you think this a good idea? Is this about
> > > boot time of the kernel image? The thing I really do not want to have is
> > > additional kernel source uploads to the archive for just those cloud
> > > kernel images, but you already considered that a bad idea (from what I
> > > read between your lines).
> > 
> > When the Google Cloud people talked to me about slow booting, it turned
> > out that reconfiguring initramfs-tools to MODULES=dep made a big
> > improvement.  That is likely to be a sensible configuration for most
> > cloud images.
> 
> I'm not sure that'll work for us. The image generation is not generally
> expected to occur on cloud instances (though in practice it certainly
> may).

That discussion was about the images provided by Google, not by Debian.
 You're quite right that it won't be suitable for images that may be
built in a different VM configuration.

> OTOH, the list of required modules may be small enough for us to
> enumerate the ones we need for booting in /etc/initramfs-tools/modules.

...and then you could use MODULES=list.  initramfs-tools will still
follow module static dependencies in this case.

> I will look into this, and we'll see what it does to boot times.

Note that the saving will mainly be in time to load the initramfs -
which on Google Compute Engine is done through BIOS disk services that
have very low performance.  The mere presence of the unneeded modules
in the initramfs won't cause them to be loaded into the kernel and
shouldn't make much difference to the time taken to boot after this
point.

Ben.

-- 
Ben Hutchings
Teamwork is essential - it allows you to blame someone else.

[toc] | [prev] | [next] | [standalone]

#58800

[Multipart message — attachments visible in raw view] — view raw

On Mon, Aug 28, 2017 at 01:31:31AM +0100, Ben Hutchings wrote:
> > OTOH, the list of required modules may be small enough for us to
> > enumerate the ones we need for booting in /etc/initramfs-tools/modules.
> 
> ...and then you could use MODULES=list.  initramfs-tools will still
> follow module static dependencies in this case.
> 
> > I will look into this, and we'll see what it does to boot times.
> 
> Note that the saving will mainly be in time to load the initramfs -
> which on Google Compute Engine is done through BIOS disk services that
> have very low performance.  The mere presence of the unneeded modules
> in the initramfs won't cause them to be loaded into the kernel and
> shouldn't make much difference to the time taken to boot after this
> point.

On Amazon's HVM instance families, the initramfs is read from "local"
disk, which may be network-attached or actually local. I haven't
profiled load times in great depth, but my guess is that reading and
uncompressing the image would be the biggest contributors to the load
time. In my experimentation, uncompressing an 18 MB initramfs takes
roughly 500 ms of clock time when read from network storage. That's not
completely insignificant, but considering the fragility of MODULES=list
or MODULES=dep, I'm not sure it's the best place to look for
optimizations right now.

noah

[toc] | [prev] | [next] | [standalone]

#58783

On 08/26/2017 12:15 PM, Martin Zobel-Helas wrote:
> a) we need to decide then if we need one kernel flavour for each cloud
> provider or if we can agree on a basic set of kernel compile options
> that every cloud provider can use.

I don't think that's needed at all.

> b) most kernels Debian ships are kernels that have most drivers needed
> as modules, so even though the kernel images are big, the kernel should
> only load modules it really needs.

Size is a concern, yes. Having too many modules slows down all sorts of
things. I'm not sure, but aren't all modules added to the initrd by default?

> Thomas, can you elaborate why you think this a good idea? Is this about
> boot time of the kernel image? The thing I really do not want to have is
> additional kernel source uploads to the archive for just those cloud
> kernel images, but you already considered that a bad idea (from what I
> read between your lines).

Basically, the only thing that I want to see is a specific config for
that kernel, nothing else. Otherwise, it's going to be too much
maintenance work. Indeed, it should *not* be a different source upload,
that's too much work as well. There also may be some optimization that
we could do.

Also, I don't see this happening without a prior agreement from the
kernel team (which means probably that Ben has to agree). On our side,
we could prepare a list of kernel modules that we do *not* want.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#58805

[Multipart message — attachments visible in raw view] — view raw

On Sun, Aug 27, 2017 at 04:16:50PM +0200, Thomas Goirand wrote:
> Basically, the only thing that I want to see is a specific config for
> that kernel, nothing else. Otherwise, it's going to be too much
> maintenance work. Indeed, it should *not* be a different source upload,
> that's too much work as well. There also may be some optimization that
> we could do.
> 
> Also, I don't see this happening without a prior agreement from the
> kernel team (which means probably that Ben has to agree). On our side,
> we could prepare a list of kernel modules that we do *not* want.

You might consider looking at what Ubuntu did to their kernel.
https://insights.ubuntu.com/2017/04/05/ubuntu-on-aws-gets-serious-performance-boost-with-aws-tuned-kernel/
suggests that they did more than just disable some modules, but it's
light on details.

If we're able to come up with a specific list of proposed changes, we'll
probably be able to have a more fruitful conversation.

noah

[toc] | [prev] | [next] | [standalone]

#58760

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2017-08-26 at 11:48 +0200, Thomas Goirand wrote:
> Dear Kernel maintainers,
> 
> As you may know, it's been years that Ubuntu is shipping a kernel
> designed for the cloud. Such a kernel is simply a version of the kernel
> that is stripped down for running on VMs. The point here is that VMs do
> not need all the drivers that we typically build for the generic Debian
> kernel (and if one still needs it, a fallback to the generic kernel is
> always possible). This makes the kernel binary package a lot smaller,
> and also potentially reduces the surface of attack in case of a security
> problem. For example, we wouldn't need ax25, appletalk and such, which
> are unfortunately automatically loaded in case matching packets are
> received by the kernel,

They aren't.

Ben.

> and which have been proven to be problematic in
> terms of security maintenance. Most hardware drivers would also go away.
> 
> Since it is only a mater of *removing* some modules, I don't think
> adding a cloud / VM kernel flavor would be a lot of maintenance. Though
> of course, as I wouldn't be the one doing it, it is not up to me to
> judge the amount of work.
> 
> Could we see this happening in Debian? Please let us know your thoughts.
> 
> Cheers,
> 
> Thomas Goirand (zigo)
> 
-- 
Ben Hutchings
One of the nice things about standards is that there are so many of
them.

[toc] | [prev] | [next] | [standalone]

#58802

On Sat, Aug 26, 2017 at 11:48:22AM +0200, Thomas Goirand wrote:
>                   This makes the kernel binary package a lot smaller,

Is the size a problem right now?

> and also potentially reduces the surface of attack in case of a security
> problem.

What attach surface?

>          For example, we wouldn't need ax25, appletalk and such, which
> are unfortunately automatically loaded in case matching packets are
> received by the kernel,

This is long gone.  And apart from OpenStack you would not receive such
packets ever anyway.

>                                Most hardware drivers would also go away.

Which ones?

> Could we see this happening in Debian? Please let us know your thoughts.

Please provide numbers what benefits such a seperate kernel package
would provide.  Apart from the raw size I'm not seeing it.

Also the addition of a different package means that we need to track
hardware requirements for the different cloud vendors pretty closely.
So please define what you think should be in it and what should be not.

Recently there have been several changes on what drivers are needed.
EC2 started to switch to virtual functions for the network devices.
Azure does this as well as option for the larges VM sizes, using large
Mellanox cards.

Other vendors may as well, but would you get the memo that something new
it needed?  Especially OpenStack runs in such diverse environments
that you may find everything.

Bastian

-- 
Immortality consists largely of boredom.
		-- Zefrem Cochrane, "Metamorphosis", stardate 3219.8

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web