Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #65942 > unrolled thread

Bug#947759: Configuration optimizations for the cloud variant

Back to article view | Back to linux.debian.kernel

Contents

  Bug#947759: Configuration optimizations for the cloud variant Josh Triplett <josh@joshtriplett.org> - 2019-12-30 09:10 +0100
    Bug#947759: Configuration optimizations for the cloud variant Ben Hutchings <ben@decadent.org.uk> - 2019-12-31 18:10 +0100
      Bug#947759: Configuration optimizations for the cloud variant Josh Triplett <josh@joshtriplett.org> - 2020-01-01 00:40 +0100
        Bug#947759: Configuration optimizations for the cloud variant Josh Triplett <josh@joshtriplett.org> - 2020-01-21 01:50 +0100
          Bug#947759: Configuration optimizations for the cloud variant Noah Meyerhans <noahm@debian.org> - 2020-01-21 03:00 +0100
            Bug#947759: Configuration optimizations for the cloud variant Noah Meyerhans <noahm@debian.org> - 2020-02-04 21:10 +0100
              Bug#947759: Configuration optimizations for the cloud variant Josh Triplett <josh@joshtriplett.org> - 2020-02-05 01:00 +0100
                Bug#947759: Configuration optimizations for the cloud variant Noah Meyerhans <noahm@debian.org> - 2020-02-05 22:40 +0100
                  Bug#947759: Configuration optimizations for the cloud variant Josh Triplett <josh@joshtriplett.org> - 2020-02-06 03:00 +0100
    Bug#947759: marked as done (Configuration optimizations for the  cloud variant) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2020-04-17 05:10 +0200

#65942 — Bug#947759: Configuration optimizations for the cloud variant

Source: linux
Severity: wishlist

A few requests for changes to the cloud configuration:

- Please disable CONFIG_ACPI_BGRT; a cloud kernel doesn't need to spend
  time or code space looking for a boot logo that won't exist.

- Please disable the CONFIG_CPU_SUP_* options for CPUs that no cloud
  provider uses.

- Please disable CONFIG_GNSS_*, which won't be hooked up to a cloud
  server.

- Please disable CONFIG_GTP for the same reason.

- Please configure CONFIG_INPUT_MOUSEDEV as a module, not built-in, as
  most cloud servers won't have a mouse and probing for one takes time.

- Please consider changing the default kernel compression to GZIP, which
  decompresses faster and thus boots faster.

- Please change CONFIG_NET_MPLS_GSO from y to m (and consider doing this
  for the non-cloud kernel too); it doesn't need to be built into the
  kernel.

- Please change CONFIG_NF_NAT_* and CONFIG_NF_MASQUERADE_* from y to m,
  as many systems won't need those modules and shouldn't need to load
  their code.

- Please compile in the NVME driver and the EXT4 filesystem; this will
  allow many cloud systems to avoid using an initramfs at all, which
  substantially improves boot time.

- Please disable CONFIG_NUMA_EMU, only used to create fake-NUMA systems
  for debugging.

- Please disable CONFIG_PCIPCWATCHDOG, CONFIG_PPS, and CONFIG_RMI4_*,
  which won't appear on a cloud server.

- Please enable CONFIG_SCHED_MC_PRIO (on both cloud and non-cloud
  kernels).

- Please disable CONFIG_VFIO_PCI_VGA and CONFIG_VGA_SWITCHEROO, which
  won't appear on cloud.

[toc] | [next] | [standalone]

#65946

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2019-12-30 at 00:03 -0800, Josh Triplett wrote:
> Source: linux
> Severity: wishlist
> 
> A few requests for changes to the cloud configuration:

Most of this looks reasonable, but:

> - Please disable CONFIG_ACPI_BGRT; a cloud kernel doesn't need to spend
>   time or code space looking for a boot logo that won't exist.
> 
> - Please disable the CONFIG_CPU_SUP_* options for CPUs that no cloud
>   provider uses.
> 
> - Please disable CONFIG_GNSS_*, which won't be hooked up to a cloud
>   server.
> 
> - Please disable CONFIG_GTP for the same reason.
> 
> - Please configure CONFIG_INPUT_MOUSEDEV as a module, not built-in, as
>   most cloud servers won't have a mouse and probing for one takes time.
> 
> - Please consider changing the default kernel compression to GZIP, which
>   decompresses faster and thus boots faster.

LZO or LZ4 could be even faster.

> - Please change CONFIG_NET_MPLS_GSO from y to m (and consider doing this
>   for the non-cloud kernel too); it doesn't need to be built into the
>   kernel.

Unfortunately nothing will load it if it's a module.  It's also a tiny
piece of code.  This might still be justifiable on the basis of
reducing the default attack surface though.

> - Please change CONFIG_NF_NAT_* and CONFIG_NF_MASQUERADE_* from y to m,
>   as many systems won't need those modules and shouldn't need to load
>   their code.

Some of those are boolean options, and I believe all of those that can
be modular already are.  You'll have to be more specific.

> - Please compile in the NVME driver and the EXT4 filesystem; this will
>   allow many cloud systems to avoid using an initramfs at all, which
>   substantially improves boot time.
> 
> - Please disable CONFIG_NUMA_EMU, only used to create fake-NUMA systems
>   for debugging.
> 
> - Please disable CONFIG_PCIPCWATCHDOG, CONFIG_PPS, and CONFIG_RMI4_*,
>   which won't appear on a cloud server.

Hyper-V and KVM both support PTP clocks, and PTP_1588_CLOCK selects
PPS.

> - Please enable CONFIG_SCHED_MC_PRIO (on both cloud and non-cloud
>   kernels).
> 
> - Please disable CONFIG_VFIO_PCI_VGA and CONFIG_VGA_SWITCHEROO, which
>   won't appear on cloud.

Ben.

-- 
Ben Hutchings
Experience is what causes a person to make new mistakes
instead of old ones.

[toc] | [prev] | [next] | [standalone]

#65950

On Tue, 31 Dec 2019 17:03:23 +0000 Ben Hutchings <ben@decadent.org.uk> wrote:
> On Mon, 2019-12-30 at 00:03 -0800, Josh Triplett wrote:
> > Source: linux
> > Severity: wishlist
> > 
> > A few requests for changes to the cloud configuration:
> 
> Most of this looks reasonable, but:
> 
> > - Please disable CONFIG_ACPI_BGRT; a cloud kernel doesn't need to spend
> >   time or code space looking for a boot logo that won't exist.
> > 
> > - Please disable the CONFIG_CPU_SUP_* options for CPUs that no cloud
> >   provider uses.
> > 
> > - Please disable CONFIG_GNSS_*, which won't be hooked up to a cloud
> >   server.
> > 
> > - Please disable CONFIG_GTP for the same reason.
> > 
> > - Please configure CONFIG_INPUT_MOUSEDEV as a module, not built-in, as
> >   most cloud servers won't have a mouse and probing for one takes time.
> > 
> > - Please consider changing the default kernel compression to GZIP, which
> >   decompresses faster and thus boots faster.
> 
> LZO or LZ4 could be even faster.

It might. GZIP seemed like a more reasonable tradeoff to request, but
I'd be all for LZ4 if that seems reasonable to you. That is indeed the
fastest option (see https://smackerelofopinion.blogspot.com/2019/09/ for
data to back that up).

It might even be worth switching to LZ4 on the standard Debian kernels
as well.

> > - Please change CONFIG_NET_MPLS_GSO from y to m (and consider doing this
> >   for the non-cloud kernel too); it doesn't need to be built into the
> >   kernel.
> 
> Unfortunately nothing will load it if it's a module.  It's also a tiny
> piece of code.  This might still be justifiable on the basis of
> reducing the default attack surface though.

I didn't realize it didn't autoload. But yes, it seems quite unlikely to
be needed.

> > - Please change CONFIG_NF_NAT_* and CONFIG_NF_MASQUERADE_* from y to m,
> >   as many systems won't need those modules and shouldn't need to load
> >   their code.
> 
> Some of those are boolean options, and I believe all of those that can
> be modular already are.  You'll have to be more specific.

I'd forgotten that several options became booleans and their code got
merged into parent modules that are already modularized; nevermind on
this one.

> > - Please compile in the NVME driver and the EXT4 filesystem; this will
> >   allow many cloud systems to avoid using an initramfs at all, which
> >   substantially improves boot time.
> > 
> > - Please disable CONFIG_NUMA_EMU, only used to create fake-NUMA systems
> >   for debugging.
> > 
> > - Please disable CONFIG_PCIPCWATCHDOG, CONFIG_PPS, and CONFIG_RMI4_*,
> >   which won't appear on a cloud server.
> 
> Hyper-V and KVM both support PTP clocks, and PTP_1588_CLOCK selects
> PPS.

Ah, I didn't realize Hyper-V provided a PTP device; that makes sense.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#66177

Following up on this, here's a simplified list of optimizations for the
cloud variant in one place, taking into account the previous reply.
Would it help to get this in the form of a patch or MR on
https://salsa.debian.org/kernel-team/linux/ ?

- Please compile in the NVME driver and the EXT4 filesystem; this will
  allow many cloud systems to avoid using an initramfs at all, which
  substantially improves boot time.

- Please consider changing the default kernel compression to LZ4, which
  decompresses much faster and thus boots faster. (Benchmarks to support
  this: https://smackerelofopinion.blogspot.com/2019/09/ ) Please
  consider doing this on the non-cloud configuration as well.

- Please disable CONFIG_ACPI_BGRT; a cloud kernel doesn't need to spend
  time or code space looking for a boot logo that won't exist.

- Please disable the CONFIG_CPU_SUP_* options for CPUs that no cloud
  provider uses (concretely, please disable everything except
  CONFIG_CPU_SUP_AMD and CONFIG_CPU_SUP_INTEL).

- Please disable CONFIG_GNSS_*, which won't be hooked up to a cloud
  server.

- Please disable CONFIG_GTP for the same reason.

- Please configure CONFIG_INPUT_MOUSEDEV as a module, not built-in, as
  most cloud servers won't have a mouse and probing for one takes time.

- Please change CONFIG_NET_MPLS_GSO from y to m. While nothing will
  automatically load it, it's sufficiently rare that loading it manually
  seems reasonable, and this would reduce attack surface and boot time.

- Please disable CONFIG_NUMA_EMU, only used to create fake-NUMA systems
  for debugging.

- Please disable CONFIG_PCIPCWATCHDOG and CONFIG_RMI4_*, which won't
  appear on a cloud server.

- Please enable CONFIG_SCHED_MC_PRIO (on both cloud and non-cloud
  kernels).

- Please disable CONFIG_VFIO_PCI_VGA and CONFIG_VGA_SWITCHEROO, which
  won't appear on cloud.

Thank you,
Josh Triplett

[toc] | [prev] | [next] | [standalone]

#66178

On Mon, Jan 20, 2020 at 04:38:55PM -0800, Josh Triplett wrote:
> Following up on this, here's a simplified list of optimizations for the
> cloud variant in one place, taking into account the previous reply.
> Would it help to get this in the form of a patch or MR on
> https://salsa.debian.org/kernel-team/linux/ ?

Hi Josh.  I started some work on integrating and testing your proposed
changes in a private branch at
https://salsa.debian.org/noahm/linux/tree/cloud-optimizations-bug947759,
but haven't gotten to the point of creating an MR yet.  I can likely
pick this up again this week, but if you create one first, that's fine
too.  One thing to note is that MR !193 changes the layout of the cloud
configs as it introduces an arm64 cloud flavour.  It hasn't been fully
reviewed yet, but there will be some merging to do if you work on your
MR independently of that.
https://salsa.debian.org/kernel-team/linux/merge_requests/193

Thank you for opening this bug!

noah

[toc] | [prev] | [next] | [standalone]

#66311

Before optimizations:

$ systemd-analyze
Startup finished in 7.828s (kernel) + 22.332s (userspace) = 30.161s 
graphical.target reached after 20.312s in userspace

With optimizations:
$ systemd-analyze
Startup finished in 1.968s (kernel) + 6.536s (userspace) = 8.504s 
graphical.target reached after 6.197s in userspace

Only tested so far on an EC2 t3.medium, so results may vary elsewhere,
but that's nice. :)

I'll post a WIP MR on salsa next.  Some cleanup is needed still.

noah

[toc] | [prev] | [next] | [standalone]

#66313

On Tue, Feb 04, 2020 at 01:40:24PM -0500, Noah Meyerhans wrote:
> Before optimizations:
> 
> $ systemd-analyze
> Startup finished in 7.828s (kernel) + 22.332s (userspace) = 30.161s 
> graphical.target reached after 20.312s in userspace
> 
> With optimizations:
> $ systemd-analyze
> Startup finished in 1.968s (kernel) + 6.536s (userspace) = 8.504s 
> graphical.target reached after 6.197s in userspace
> 
> Only tested so far on an EC2 t3.medium, so results may vary elsewhere,
> but that's nice. :)

Very nice!

I would suggest testing on a c5.large. t2 and t3 have shared CPUs, so
they have less consistent boot time. c5.large is about the same cost as
t3.large, but will have far more consistent performance.

> I'll post a WIP MR on salsa next.  Some cleanup is needed still.

Thank you for working on this!

Have you confirmed that with the optimizations, you can boot without
needing an initramfs?

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#66317

On Tue, Feb 04, 2020 at 03:48:32PM -0800, Josh Triplett wrote:
> I would suggest testing on a c5.large. t2 and t3 have shared CPUs, so
> they have less consistent boot time. c5.large is about the same cost as
> t3.large, but will have far more consistent performance.

Performance definitely seems to vary quite a bit.  Here are a few
samples from c5.large instances in us-west-2:

5.5 "generic" kernel:
$ systemd-analyze 
Startup finished in 4.323s (kernel) + 11.189s (userspace) = 15.513s 
graphical.target reached after 9.865s in userspace

5.5 "cloud" with optimizations:
$ systemd-analyze 
Startup finished in 7.273s (kernel) + 21.984s (userspace) = 29.258s 
graphical.target reached after 19.811s in userspace

$ systemd-analyze 
Startup finished in 4.319s (kernel) + 13.684s (userspace) = 18.003s 
graphical.target reached after 12.321s in userspace

$ systemd-analyze 
Startup finished in 3.327s (kernel) + 9.846s (userspace) = 13.174s 
graphical.target reached after 8.831s in userspace

The optimized timings are all taken from the initial boot of newly
launched instances.

It's certainly possible that things will look better with more data, but
it's not clear yet that this change is an improvement in all cases.

> > I'll post a WIP MR on salsa next.  Some cleanup is needed still.
> 
> Thank you for working on this!
> 

The MR is here: https://salsa.debian.org/kernel-team/linux/merge_requests/206

I'm happy to share binary and/or source .debs and/or AMIs if you'd like
to run some of your own tests.

> Have you confirmed that with the optimizations, you can boot without
> needing an initramfs?

I've confirmed that the kernel can boot on c5 and t3 instances without
an initramfs present at all.  However, the Xen backed EC2 instance types
would also need to statically link (at least) ATA_PIIX, ATA_GENERIC, and
XEN_BLKDEV_FRONTEND if we wanted to provide kernels that could be
generally useful in EC2 without an initramfs.

noah

[toc] | [prev] | [next] | [standalone]

#66319

On Wed, Feb 05, 2020 at 04:27:23PM -0500, Noah Meyerhans wrote:
> On Tue, Feb 04, 2020 at 03:48:32PM -0800, Josh Triplett wrote:
> > I would suggest testing on a c5.large. t2 and t3 have shared CPUs, so
> > they have less consistent boot time. c5.large is about the same cost as
> > t3.large, but will have far more consistent performance.
> 
> Performance definitely seems to vary quite a bit.  Here are a few
> samples from c5.large instances in us-west-2:
> 
> 5.5 "generic" kernel:
> $ systemd-analyze 
> Startup finished in 4.323s (kernel) + 11.189s (userspace) = 15.513s 
> graphical.target reached after 9.865s in userspace
> 
> 5.5 "cloud" with optimizations:
> $ systemd-analyze 
> Startup finished in 7.273s (kernel) + 21.984s (userspace) = 29.258s 
> graphical.target reached after 19.811s in userspace
> 
> $ systemd-analyze 
> Startup finished in 4.319s (kernel) + 13.684s (userspace) = 18.003s 
> graphical.target reached after 12.321s in userspace
> 
> $ systemd-analyze 
> Startup finished in 3.327s (kernel) + 9.846s (userspace) = 13.174s 
> graphical.target reached after 8.831s in userspace
> 
> The optimized timings are all taken from the initial boot of newly
> launched instances.
> 
> It's certainly possible that things will look better with more data, but
> it's not clear yet that this change is an improvement in all cases.

That's strange. I've gotten very consistent boot-time results from c5
instances, and nowhere near those values. Certainly none of these
changes should introduce anywhere near that much variability in boot
time.

At the moment, I'm focused on the kernel time, not userspace time, since
the latter depends greatly on what you have installed. Would you mind
adding `initcall_debug` to the kernel command line, and providing
(compressed) dmesg logs for a few boots where you're observing that much
variability? That should make it relatively easy to diagnose where the
time is going.

(Also, you may want to make sure your kernel isn't configured to send
all its messages to the (virtual) serial port, which can slow down boot
time.)

> > > I'll post a WIP MR on salsa next.  Some cleanup is needed still.
> > 
> > Thank you for working on this!
> > 
> 
> The MR is here: https://salsa.debian.org/kernel-team/linux/merge_requests/206
> 
> I'm happy to share binary and/or source .debs and/or AMIs if you'd like
> to run some of your own tests.
> 
> > Have you confirmed that with the optimizations, you can boot without
> > needing an initramfs?
> 
> I've confirmed that the kernel can boot on c5 and t3 instances without
> an initramfs present at all.  However, the Xen backed EC2 instance types
> would also need to statically link (at least) ATA_PIIX, ATA_GENERIC, and
> XEN_BLKDEV_FRONTEND if we wanted to provide kernels that could be
> generally useful in EC2 without an initramfs.

For initramfs-less boot, I would suggest focusing on AWS's new
Nitro-based instances, rather than the Xen-based instances. I don't
think there's value in trying to build in enough drivers to boot without
an initramfs on older cloud hardware as well.

[toc] | [prev] | [next] | [standalone]

#66785 — Bug#947759: marked as done (Configuration optimizations for the cloud variant)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Fri, 17 Apr 2020 03:00:10 +0000
with message-id <E1jPHEg-000DpB-5M@fasolo.debian.org>
and subject line Bug#947759: fixed in linux 5.5.17-1
has caused the Debian Bug report #947759,
regarding Configuration optimizations for the cloud variant
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
947759: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=947759
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web