Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1570354 > unrolled thread

Re: [PATCH 01/50] x86/boot/e820: Introduce arch/x86/include/asm/e820/types.h

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Sam Ravnborg <sam@ravnborg.org> - 2017-01-31 06:50 +0100
    Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Ingo Molnar <mingo@kernel.org> - 2017-01-31 17:50 +0100
      Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Sam Ravnborg <sam@ravnborg.org> - 2017-01-31 18:30 +0100
        Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Joe Perches <joe@perches.com> - 2017-01-31 19:10 +0100
        Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Ingo Molnar <mingo@kernel.org> - 2017-01-31 19:10 +0100
          Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Sam Ravnborg <sam@ravnborg.org> - 2017-01-31 20:30 +0100
            Re: [PATCH 01/50] x86/boot/e820: Introduce  arch/x86/include/asm/e820/types.h Ingo Molnar <mingo@kernel.org> - 2017-02-01 10:00 +0100

#1570354 — Re: [PATCH 01/50] x86/boot/e820: Introduce arch/x86/include/asm/e820/types.h

On Mon, Jan 30, 2017 at 08:58:33AM +0100, Ingo Molnar wrote:
> 
> * Sam Ravnborg <sam@ravnborg.org> wrote:
> 
> > On Sat, Jan 28, 2017 at 11:11:22PM +0100, Ingo Molnar wrote:
> > > 
> > > The plan is to keep the old UAPI header in place but the kernel won't
> > > use it anymore - and after some time we'll try to remove it. (User-space
> > > tools better have local copies of headers anyway, instead of relying
> > > on kernel headers.)
> > 
> > The idea with uapi is the the kernel provides a sane set of headers
> > to be used by user space.
> > So we avoid random copies that is maintained by random people in random
> > ways resulting in random bugs.
> 
> Your argument is simplistic which presents a false dichotomy: maintaining a copy 
> or fully sharing the header are not the only two options available to share the 
> information in the headers between the kernel and tooling: for example perf uses a 
> half-automated method where headers are copied from the kernel, but also checked 
> automatically against the upstream kernel, and a (non-fatal) warning is emitted 
> during the build if the upstream header has changed.

Obvious when there are two vastly different solutions there are some
middle ground solutions too.
And perf implements a tool based method to check that the manually
copied/created/maintained headers are in sync with the kernel.
which is just another way to maintain a set of copied headers.

But this does not at all address the point which is that it is
an unessesary burden to put on all the _users_ of the kernel uapi
headers to amintain their own copy of the kernel headers
(tool assisted or not).

The rationale behind requesting the users of the kernel uapi headers
seems to be that the kernel people would like to have the freedom
to fix wrong doings in the past without violatign the uapi.
So the benefit here is simpler headers on the kernel side compared
to all users of kernel uapi heders to manually (tool assisted or not)
maintaining their own copies of the uapi kernel headers.

Heck - if this is how it should be done then the the kernel
should provide the copy of the headers that describe the
uapi and the kernel should have the tool infrastructure to check
that the headers are OK.
Not all the users outside the kernelspace.

So just to repeat - it is an error prone design to let users
of the kernel uapi maintain their own copies of the kernel
uapi header. It is the job of the kernel.

	Sam

[toc] | [next] | [standalone]

#1570891

* Sam Ravnborg <sam@ravnborg.org> wrote:

> Obvious when there are two vastly different solutions there are some
> middle ground solutions too.
>
> And perf implements a tool based method to check that the manually 
> copied/created/maintained headers are in sync with the kernel. which is just 
> another way to maintain a set of copied headers.
> 
> But this does not at all address the point which is that it is an unessesary 
> burden to put on all the _users_ of the kernel uapi headers to amintain their 
> own copy of the kernel headers (tool assisted or not).

In 99% of the cases it's trivial to keep them updated on the tooling side. 
Basically a copy only has to be made if a change is done - and changes are not 
that common in the ABI space.

> The rationale behind requesting the users of the kernel uapi headers seems to be 
> that the kernel people would like to have the freedom to fix wrong doings in the 
> past without violatign the uapi.
>
> So the benefit here is simpler headers on the kernel side compared
> to all users of kernel uapi heders to manually (tool assisted or not)
> maintaining their own copies of the uapi kernel headers.

I was reacting to this original claim of yours:

> > > The idea with uapi is the the kernel provides a sane set of headers to be 
> > > used by user space.
> > >
> > > So we avoid random copies that is maintained by random people in random ways 
> > > resulting in random bugs.

There's no 'random copies' - they are copies of past, valid versions of the kernel 
headers, and there's a mechanism to update them to the latest.

> Heck - if this is how it should be done then the the kernel
> should provide the copy of the headers that describe the
> uapi and the kernel should have the tool infrastructure to check
> that the headers are OK.
> Not all the users outside the kernelspace.
> 
> So just to repeat - it is an error prone design to let users
> of the kernel uapi maintain their own copies of the kernel
> uapi header. It is the job of the kernel.

But "random copies" is not what perf does. Tell me, how is the perf mechanism of 
using the headers "error-prone"? It's a delayed COW mechanism - COW is not an 
error-prone concept in any way ...

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1570921

> > 
> > So just to repeat - it is an error prone design to let users
> > of the kernel uapi maintain their own copies of the kernel
> > uapi header. It is the job of the kernel.
> 
> But "random copies" is not what perf does. Tell me, how is the perf mechanism of 
> using the headers "error-prone"? It's a delayed COW mechanism - COW is not an 
> error-prone concept in any way ...

The whole concept that user space have the burden to maintain
a set of headers describing the uapi provided by the kernel is the point
of discussion.

The randomness come into play when a user space developer are
faced with the challenge that the programm require access to something
described by the kernel uapi and then have to hunt for a header
that describes said uapi.

In this thread we have covered one rational reason to push thus
burden to user space - to give the kernel the freedom to repair
past stupidity (being that in naming or some other sort).

So lets turn around the arguments - and from a user space
perspective what is the benefit of maintaining a set of headers
describing the kernel uapi?

Obviously this allows user space to name thing exactly the
way they like, and allows user space to put all sorts of strange
things in the header files describing the kernel uapi.

This is just not enough good reasons why the user space
developer shall create headers files describing
the kerneluapi and maintain tooling to maintain the header
files describing the kernel uapi.

Are there other benefits that is missed which makes the
concept of letting user space maintain header files describing
the kernel uapi a good idea that is missed?

	Sam

[toc] | [prev] | [next] | [standalone]

#1570955

On Tue, 2017-01-31 at 19:00 +0100, Ingo Molnar wrote:
> * Sam Ravnborg <sam@ravnborg.org> wrote:
> > In this thread we have covered one rational reason to push thus
> > burden to user space - to give the kernel the freedom to repair
> > past stupidity (being that in naming or some other sort).
> 
> There's no real "burden" for heaven's sake: it's having to execute a 'cp' every 
> now and then and check whether the result still builds (it will build just fine in 
> the overwhelming majority of cases).

Defining what is it that encompasses "userspace", the
thing that's nominally guaranteed to "not be broken",
here is the real question.

Is it just the ABI or is it more than that?

[toc] | [prev] | [next] | [standalone]

#1570959

* Sam Ravnborg <sam@ravnborg.org> wrote:

> > > 
> > > So just to repeat - it is an error prone design to let users
> > > of the kernel uapi maintain their own copies of the kernel
> > > uapi header. It is the job of the kernel.
> > 
> > But "random copies" is not what perf does. Tell me, how is the perf mechanism of 
> > using the headers "error-prone"? It's a delayed COW mechanism - COW is not an 
> > error-prone concept in any way ...
> 
> The whole concept that user space have the burden to maintain a set of headers 
> describing the uapi provided by the kernel is the point of discussion.

That's your characterisation and I think it's misleading: they don't have to 
"maintain a set of headers", they can, if they'd like to:

   "maintain a _copy_ of a small number of headers that the typical user-space 
    tooling project is interested in"

which is different from how you framed it.

> The randomness come into play when a user space developer are faced with the 
> challenge that the programm require access to something described by the kernel 
> uapi and then have to hunt for a header that describes said uapi.
>
> In this thread we have covered one rational reason to push thus
> burden to user space - to give the kernel the freedom to repair
> past stupidity (being that in naming or some other sort).

There's no real "burden" for heaven's sake: it's having to execute a 'cp' every 
now and then and check whether the result still builds (it will build just fine in 
the overwhelming majority of cases).

> So lets turn around the arguments - and from a user space
> perspective what is the benefit of maintaining a set of headers
> describing the kernel uapi?

Firsty, the headers are not maintained by the user-space project, 99.999% of the 
maintenance is done by the kernel developers.

> Obviously this allows user space to name thing exactly the way they like, and 
> allows user space to put all sorts of strange things in the header files 
> describing the kernel uapi.

Huh? Again, my suggestion is to to _copy_ the kernel header the tooling project is 
interested in as-is, and this is exactly what perf does. We don't change the UAPI 
headers, we copy them verbatim and check during the build that the copy is exact.

> Are there other benefits that is missed which makes the
> concept of letting user space maintain header files describing
> the kernel uapi a good idea that is missed?

Yes, you missed a lot of the benefits.

Firstly, the user-space tooling project that relies on some UAPI header with Linux 
kernel ABI details in it, if it so wishes, maintains a _copy_ of the affected 
headers, which is vastly less work and 'burden' than 'maintaining headers'.

Secondly, doing this copy has numerous advantages over using the UAPI headers from 
the distro installation:

 - Additions to the ABI are monitored by the tooling project. Say there's 
   external tooling that relies on the perf system call, and uses a copy of 
   perf_events.h. It can see new ABI additions in new versions of the file, and 
   can add support for them. If on the other hand the file is just shared with 
   whatever distro kernel headers there are on the system, there's no information 
   about ABI progress.

   Advantage: changes to the ABI can be seen in the tool's Git repository,
              the "git log" of the header shows how the versions got updated and 
              how the tool adopted to new ABI additions.

 - The tooling project is more self-contained: it can build just fine in more 
   limited environments that don't have the kernel headers installed.

   Advantage: more robust tool build process, fewer dependencies.

 - So the tooling project can copy the kernel header _before_ the updated kernel
   header trickles through to distros. This has the advantage of being able to add
   tooling support for new ABI additions before it all goes into the 'official'
   UAPI headers a distro includes, and speeds up ABI propagation.

   Advantage: faster ABI propagation to tooling.

 - In fact this model allows the tooling project to host new kernel ABI additions 
   in its development process, so sometimes it has the new UAPI header _before_ 
   the upstream kernel has it. (Obviously the upstream kernel has to pull it for 
   it to become the official UAPI header, so this only works for subsystems where
   there's good communication between the tooling developers and the kernel 
   developers.)

   This is what tools/perf does quite often.

   Advantage: better development model where tooling can develop new kernel 
              features hand in hand with kernel developers.

I.e. most of the advantages of the copied headers are for the _tooling project_, 
not for the kernel.

Your one sided characterization of the copying method as a 'burden' ignores all 
that.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1571024

Hi Ingo.

> >
> > In this thread we have covered one rational reason to push thus
> > burden to user space - to give the kernel the freedom to repair
> > past stupidity (being that in naming or some other sort).
> 
> There's no real "burden" for heaven's sake: it's having to execute a 'cp' every 
> now and then and check whether the result still builds (it will build just fine in 
> the overwhelming majority of cases).

We are obvious so far away in our perception of what is easy for
a user-space developers that is is not even funny.

> > So lets turn around the arguments - and from a user space
> > perspective what is the benefit of maintaining a set of headers
> > describing the kernel uapi?
> 
> Firsty, the headers are not maintained by the user-space project, 99.999% of the 
> maintenance is done by the kernel developers.

In the inital mail triggering this plan was that the kernel
is moving away from having uapi headers what-so-ever.

Quoting the original mail:
"
The plan is to keep the old UAPI header in place but the kernel won't
use it anymore - and after some time we'll try to remove it. 
"

Translated:
The plan is that the kernel will stop using headers from uapi/*
The headers will be left for a while and then they will be deleted.

The mail was centered about e820 - so maybe the outlined plan
was only for e820.
But I read it as a general pan - hence this mail thread.

So this is the plan that this argument is about.
No tooling will magically make files appear again.
And there is no benefit from user space that the kernel remove the files.

And the removing of files from uapi/ makes is hard for user space.
Because then a user-space developers have to find a definition
of the uapi somewhere else.

> Huh? Again, my suggestion is to to _copy_ the kernel header the tooling project is 
> interested in as-is, and this is exactly what perf does.
Here the proposal was stop using the header in uapi/ and then later delete it.
So there i no file to copy - making the copying and tooling part irrelevant.

> Yes, you missed a lot of the benefits.
> 
> Firstly, the user-space tooling project that relies on some UAPI header with Linux 
> kernel ABI details in it, if it so wishes, maintains a _copy_ of the affected 
> headers, which is vastly less work and 'burden' than 'maintaining headers'.

It is by no means a benefit to have a copy, rather than using the source.
There is a reason why we do not have multiple copies of the headers in the kernel
when we can avoid it. asm-generic is one way the kernel avoid the burden
of maintaining copies of headers.

All your remaining argumens zapped - I see the point of view.
But there are many other solutions for the same set of problems.

Perf being intimidate with the kernel is not the best example to come up with.
Think about to 100's of program that uses a few ioclt to talk with drivers etc.

	Sam

[toc] | [prev] | [next] | [standalone]

#1571334

* Sam Ravnborg <sam@ravnborg.org> wrote:

> > Firsty, the headers are not maintained by the user-space project, 99.999% of 
> > the maintenance is done by the kernel developers.
> 
> In the inital mail triggering this plan was that the kernel is moving away from 
> having uapi headers what-so-ever.

No, that is a misunderstanding:

> Quoting the original mail:
> "
> The plan is to keep the old UAPI header in place but the kernel won't
> use it anymore - and after some time we'll try to remove it. 
> "

You misunderstood my mail and you misunderstood the patch: we transition from the 
old UAPI header to a new one, but the exported data structures are still kept!

If you check the patches you'll see that bootparam.h still exports the e820_entry 
data structure. The 'old' header is simply one that is being phased out (if we 
can) - but the information is still exported.

> Translated:
> The plan is that the kernel will stop using headers from uapi/*
> The headers will be left for a while and then they will be deleted.

No, not at all.

> Perf being intimidate with the kernel is not the best example to come up with. 

No, that's wrong too, most larger tooling projects that care about feature 
propagation latency in fact already do something quite similar to what perf does.

For example the tooling side of GPU drivers (libdrm) has a copy of all the 
relevant UAPI headers:

triton:~/libdrm/include/drm> ls -l
total 316
-rw-rw-r-- 1 mingo mingo 19119 Feb  1 09:47 amdgpu_drm.h
-rw-rw-r-- 1 mingo mingo 11850 Feb  1 09:47 drm_fourcc.h
-rw-rw-r-- 1 mingo mingo 27613 Feb  1 09:47 drm.h
-rw-rw-r-- 1 mingo mingo 18313 Feb  1 09:47 drm_mode.h
-rw-rw-r-- 1 mingo mingo  2701 Feb  1 09:47 drm_sarea.h
-rw-rw-r-- 1 mingo mingo 46684 Feb  1 09:47 i915_drm.h
-rw-rw-r-- 1 mingo mingo  7895 Feb  1 09:47 mach64_drm.h
-rw-rw-r-- 1 mingo mingo 12923 Feb  1 09:47 mga_drm.h
-rw-rw-r-- 1 mingo mingo  5662 Feb  1 09:47 nouveau_drm.h
-rw-rw-r-- 1 mingo mingo  4217 Feb  1 09:47 qxl_drm.h
-rw-rw-r-- 1 mingo mingo  9901 Feb  1 09:47 r128_drm.h
-rw-rw-r-- 1 mingo mingo 38509 Feb  1 09:47 radeon_drm.h
-rw-rw-r-- 1 mingo mingo  5201 Feb  1 09:47 README
-rw-rw-r-- 1 mingo mingo  7054 Feb  1 09:47 savage_drm.h
-rw-rw-r-- 1 mingo mingo  2534 Feb  1 09:47 sis_drm.h
-rw-rw-r-- 1 mingo mingo  5526 Feb  1 09:47 tegra_drm.h
-rw-rw-r-- 1 mingo mingo  9534 Feb  1 09:47 vc4_drm.h
-rw-rw-r-- 1 mingo mingo  8291 Feb  1 09:47 via_drm.h
-rw-rw-r-- 1 mingo mingo  4704 Feb  1 09:47 virtgpu_drm.h
-rw-rw-r-- 1 mingo mingo 31225 Feb  1 09:47 vmwgfx_drm.h

For example i915_drm.h is a copy of include/uapi/drm/i915_drm.h, which is being 
synched between the two projects regularly.

> Think about to 100's of program that uses a few ioclt to talk with drivers etc.

Those can use distro UAPI headers just fine, if they don't care about the 6-12 
months delay it takes to get updated kernel headers. I.e. what you propose works 
for well-established ABIs that have been around for years.

To actually _progress_ with a tooling project, in close cooperation with the 
kernel side, the UAPI method of sharing via distro headers as-is hinders 
development agility big time...

Distro UAPI headers work fine in a world where the kernel is a static entity and 
does not update its ABIs. I.e. it only works if there's no actual kernel side 
extensions to the ABI. The whole UAPI distro headers approach is designed for the 
case where the style of sharing the headers matters the least: for a stagnant 
kernel or a stagnant tooling project ...

Btw., his kind of rigid, suboptimal, latency laden method of sharing information 
between the kernel and tooling might be one of the reasons why in general the 
Linux tooling landscape sucks, compared to other OSs...

Thanks,

	Ingo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web