Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1464737 > unrolled thread

[RFC PATCH 0/5] Add CONFIG symbol as module attribute

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH 0/5] Add CONFIG symbol as module attribute Cristina Moraru <cristina.moraru09@gmail.com> - 2016-08-17 21:30 +0200
    [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter Cristina Moraru <cristina.moraru09@gmail.com> - 2016-08-17 21:30 +0200
      Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation  parameter "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-19 03:00 +0200
        Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation  parameter "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-19 03:40 +0200
        Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter Cristina-Gabriela Moraru <cristina.moraru09@gmail.com> - 2016-08-20 17:20 +0200
          Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation  parameter "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-23 21:20 +0200
    [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute Cristina Moraru <cristina.moraru09@gmail.com> - 2016-08-17 21:30 +0200
      Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb  module attribute "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-19 04:10 +0200
        Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb  module attribute Cristina-Gabriela Moraru <cristina.moraru09@gmail.com> - 2016-08-20 17:20 +0200
          Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb  module attribute "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-23 21:20 +0200
    [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile Cristina Moraru <cristina.moraru09@gmail.com> - 2016-08-17 21:30 +0200
      Re: [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-18 21:50 +0200
    Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-19 04:10 +0200
      Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Michal Marek <mmarek@suse.com> - 2016-08-19 11:10 +0200
        Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Cristina-Gabriela Moraru <cristina.moraru09@gmail.com> - 2016-08-22 21:50 +0200
        Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-23 23:40 +0200
          Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Michal Marek <mmarek@suse.com> - 2016-08-24 13:10 +0200
            Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-24 18:40 +0200
              Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Naveen Kumar <nk1383886@gmail.com> - 2016-08-24 19:40 +0200
      Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Cristina-Gabriela Moraru <cristina.moraru09@gmail.com> - 2016-08-22 21:40 +0200
        Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-08-23 21:20 +0200
      Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Christoph Hellwig <hch@lst.de> - 2016-08-25 09:50 +0200
        Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Johannes Berg <johannes@sipsolutions.net> - 2016-08-25 10:40 +0200
        Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute Michal Marek <mmarek@suse.com> - 2016-08-25 10:50 +0200

Page 1 of 2  [1] 2  Next page →

#1464737 — [RFC PATCH 0/5] Add CONFIG symbol as module attribute

This patchset implements dynamic pegging of kconfig symbol
into driver modinfo section

* updates streamline_config.pl to generate the auxiliary file
scripts/mod/Module.ksymb containing associations of driver file
names and corresponding kconfig symbols CONFIG_*
* updates Makefiles to trigger streamline_config.pl call for
Module.ksymb generation and pass the determined CONFIG_* symbol
as compilation parameter -D in KBUILD_KSYMB variable
* adds kconfig_symb as module attribute
* updates modpost to set KBUILD_KSYMB macro as value for
kbuild_symb attribute

Note: the content of the file Module.ksymb is generated for
all modules that have only one associate CONFIG option. All
others are considered to be components linked at the final
modules but not final modules itselves.

The result of this patchset is the following. All modules from
/sys expose the correct CONFIG_* symbol in the module attribute
kconfig_symb with some exceptions. For a total number of 58
modules, 4 of them do not:

snd_seq_midi_event
mptscsih
libahci
mptbase

After a short research:

For mptscsih - ./drivers/message/fusion/Makefile
obj-$(CONFIG_FUSION_SPI)	+= mptbase.o mptscsih.o mptspi.o
obj-$(CONFIG_FUSION_FC)		+= mptbase.o mptscsih.o mptfc.o
obj-$(CONFIG_FUSION_SAS)	+= mptbase.o mptscsih.o mptsas.o
As appears in the Makefile mptscsi is part of more config
options so it's excluded because it's not considered a module
by itself but, somehow gets compiled as module. Same for the
mptbase. I still have to understand better the Kbuild logic so
I cannot state that this is loose practice.

For libahci there are multiple CONFIG_ symbols for different
architectures.

Although past discussion recommended not to use streamline_config,
this patchset still relies on it, for the sake of the proof of
concept. Although the result is close to the target it still does
not provide complete correctness. However it can be replaced by
creating another script which tries to determine better the
module <-> CONFIG associations and output them in auxiliary file
Module.ksymb. Maybe this way we could also determine all CONFIGs
for a particular driver, not only the exact one that enables it.

The auxiliary file is necessary because the Makefile itself does
not have a mapping. The makefile includes the config file with
directive include which creates a series of internal variables
CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
other Makefiles, lines like 

obj-$(CONFIG_FOO) = foo.o 

are resolved in place to obj-y, obj-m etc according to 'make'
logic and the association is lost.

Finally, what does this patchset provide is an infrastructure
to dinamically peg CONFIG_* options to associate drivers using
the mapping from Module.ksymb file. Generation of Module.ksymb
can be replaced but keeping the same format permit the usage of
the other patches.

This patchset is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Cristina Moraru (5):
  Add generation of Module.symb in streamline_config
  Add CONFIG symbol to module as compilation parameter
  Trigger Module.ksymb generation in Makefile
  Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
  Add kconf_symb as kernel module attribute

 Makefile                             |  4 ++++
 include/linux/module.h               |  1 +
 kernel/module.c                      |  2 ++
 scripts/Makefile.lib                 |  9 ++++++++-
 scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
 scripts/mod/modpost.c                |  7 +++++++
 6 files changed, 51 insertions(+), 2 deletions(-)

-- 
2.7.4

[toc] | [next] | [standalone]

#1464740 — [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

Add CONFIG symbol to kernel modules as a define via -D
compilation parameter. The CONFIG_FOO symbol for each
module is determined by the module name, using the
associations from Module.ksymb. This file is loaded
using the 'include' directive, thus run like a regular
makefile. The format of the content of the file is the
following:

foo_KCONF=CONFIG_FOO

creating a set of Makefile variables foo_KCONF with the
CONFIG_FOO as values.

The Makefile then adds this value in the compilation
command with -DKBUILD_KCONF='"CONFIG_FOO"'.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 scripts/Makefile.lib | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index e7df0f5..8ae9b7f 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
 subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
 obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
 
+# Include Module.ksymb which contains the associations of modules' names
+# and corresponding CONFIG_* options
+include Module.ksymb
+
 # These flags are needed for modversions and compiling, so we define them here
 # already
 # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
@@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
 basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
 modname_flags  = $(if $(filter 1,$(words $(modname))),\
                  -DKBUILD_MODNAME=$(call name-fix,$(modname)))
+ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
+ksymb_flags = $(if $(filter 1,$(words $(modname))),\
+                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
 
 orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
                  $(ccflags-y) $(CFLAGS_$(basetarget).o)
@@ -162,7 +169,7 @@ endif
 
 c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__c_flags) $(modkern_cflags)                           \
-		 $(basename_flags) $(modname_flags)
+		 $(basename_flags) $(modname_flags) $(ksymb_flags)
 
 a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__a_flags) $(modkern_aflags)
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1465692 — Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> Add  CONFIG symbol to kernel modules as a define via -D

Perhaps better worded as:

When modules have a direct Kconfig CONFIG_ symbol associated with
we want to be able to make it available to the build system when we
are building the module.

You can then describe you do this with -D.

> compilation parameter. The CONFIG_FOO symbol for each
> module is determined by the module name, using the
> associations from Module.ksymb. This file is loaded
> using the 'include' directive, thus run like a regular
> makefile. The format of the content of the file is the
> following:
> 
> foo_KCONF=CONFIG_FOO
> 
> creating a set of Makefile variables foo_KCONF with the
> CONFIG_FOO as values.
> 
> The Makefile then adds this value in the compilation
> command with -DKBUILD_KCONF='"CONFIG_FOO"'.

Very nice. What would it mean if it lacks this ?
This should be explained on the commit log.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out as its part of the cover letter and
not relevant to the patch. You can however mention you will
use this later to make the define more useful in userspace.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/Makefile.lib | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index e7df0f5..8ae9b7f 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
>  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
>  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
>  
> +# Include Module.ksymb which contains the associations of modules' names
> +# and corresponding CONFIG_* options
> +include Module.ksymb

Is this file always generated? If not prefixing the include call with - would
be better. If we are going to make this optional, then an ifdef wrapper would
suffice as we know it'd be expected only if the new feature was enabled.

> +
>  # These flags are needed for modversions and compiling, so we define them here
>  # already
>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))

Are clashes possible with this formula? Can we end up with two results for instance?
If not what prevents current konfig logic and namespace from a clash ? If we do
not have anything to prevent a clash, what can we do to help make such clash not
possible ?

>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
> @@ -162,7 +169,7 @@ endif
>  
>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__c_flags) $(modkern_cflags)                           \
> -		 $(basename_flags) $(modname_flags)
> +		 $(basename_flags) $(modname_flags) $(ksymb_flags)
>  
>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>  		 $(__a_flags) $(modkern_aflags)
> -- 

Other than that looks good!

  Luis

[toc] | [prev] | [next] | [standalone]

#1465806 — Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

On Thu, Aug 18, 2016 at 08:10:07PM +0200, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> 
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/Makefile.lib | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> > index e7df0f5..8ae9b7f 100644
> > --- a/scripts/Makefile.lib
> > +++ b/scripts/Makefile.lib
> > @@ -89,6 +89,10 @@ multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
> >  subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
> >  obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
> >  
> > +# Include Module.ksymb which contains the associations of modules' names
> > +# and corresponding CONFIG_* options
> > +include Module.ksymb
> 
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

The ordering of the patches 2 and 3 seem to be backward, you should first
generate the file otherwise this patch will break the build as the file
is not present.

> > +
> >  # These flags are needed for modversions and compiling, so we define them here
> >  # already
> >  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> > @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> > +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> > +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> > +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> 
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

To help test this I've added your patches to a development branch on my linux-next
tree to see if 0-day picks up any issues.

Guenter, to be even more thorough can I trouble you for a spin of this branch?

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160818-gsoc-kconf_symb

Results should help give us an idea of troubling areas if this were to be added
as a new Kconfig feature even for R&D.

  Luis

[toc] | [prev] | [next] | [standalone]

#1466818 — Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
>> Add  CONFIG symbol to kernel modules as a define via -D
>
> Perhaps better worded as:
>
> When modules have a direct Kconfig CONFIG_ symbol associated with
> we want to be able to make it available to the build system when we
> are building the module.
>
> You can then describe you do this with -D.
>
>> compilation parameter. The CONFIG_FOO symbol for each
>> module is determined by the module name, using the
>> associations from Module.ksymb. This file is loaded
>> using the 'include' directive, thus run like a regular
>> makefile. The format of the content of the file is the
>> following:
>>
>> foo_KCONF=CONFIG_FOO
>>
>> creating a set of Makefile variables foo_KCONF with the
>> CONFIG_FOO as values.
>>
>> The Makefile then adds this value in the compilation
>> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
>
> Very nice. What would it mean if it lacks this ?
> This should be explained on the commit log.
>

If we lack it then KBUILD_KCONF="" and in /sys the module has the
kconfig_symbol attribute but with the empty string as content:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

I will add it into the commit log.

>> This patch is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> You can leave this out as its part of the cover letter and
> not relevant to the patch. You can however mention you will
> use this later to make the define more useful in userspace.
>
>> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
>> ---
>>  scripts/Makefile.lib | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>> index e7df0f5..8ae9b7f 100644
>> --- a/scripts/Makefile.lib
>> +++ b/scripts/Makefile.lib
>> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
>>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
>>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
>>
>> +# Include Module.ksymb which contains the associations of modules' names
>> +# and corresponding CONFIG_* options
>> +include Module.ksymb
>
> Is this file always generated? If not prefixing the include call with - would
> be better. If we are going to make this optional, then an ifdef wrapper would
> suffice as we know it'd be expected only if the new feature was enabled.

Yes. This file is always generated. In case a 'git pull' happened
between two 'make' commands, the associations should be updated. Ok. I
will add a ifdef.

>
>> +
>>  # These flags are needed for modversions and compiling, so we define them here
>>  # already
>>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
>> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
>>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
>>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
>>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
>> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
>> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
>> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
>
> Are clashes possible with this formula? Can we end up with two results for instance?
> If not what prevents current konfig logic and namespace from a clash ? If we do
> not have anything to prevent a clash, what can we do to help make such clash not
> possible ?

I think the fact that modname is unique prevents from having clashes.
KBUILD_KSYMB is found according to modname.
Currently there is no clash because I enforced the Module.ksymb to
have 1-to-1 mapping.
The only potential clash I can imagine right now it having the same
module name but architecture specific CONFIG_* symbol. If there is a
1-to-1 mapping in Module.ksymb there should be no clash in the
makefile.

>
>>  orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
>>                   $(ccflags-y) $(CFLAGS_$(basetarget).o)
>> @@ -162,7 +169,7 @@ endif
>>
>>  c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__c_flags) $(modkern_cflags)                           \
>> -              $(basename_flags) $(modname_flags)
>> +              $(basename_flags) $(modname_flags) $(ksymb_flags)
>>
>>  a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
>>                $(__a_flags) $(modkern_aflags)
>> --
>
> Other than that looks good!
>
>   Luis

[toc] | [prev] | [next] | [standalone]

#1468791 — Re: [RFC PATCH 2/5] Add CONFIG symbol to module as compilation parameter

On Sat, Aug 20, 2016 at 05:11:37PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:10 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> > On Wed, Aug 17, 2016 at 09:27:00PM +0200, Cristina Moraru wrote:
> >> Add  CONFIG symbol to kernel modules as a define via -D
> >
> > Perhaps better worded as:
> >
> > When modules have a direct Kconfig CONFIG_ symbol associated with
> > we want to be able to make it available to the build system when we
> > are building the module.
> >
> > You can then describe you do this with -D.
> >
> >> compilation parameter. The CONFIG_FOO symbol for each
> >> module is determined by the module name, using the
> >> associations from Module.ksymb. This file is loaded
> >> using the 'include' directive, thus run like a regular
> >> makefile. The format of the content of the file is the
> >> following:
> >>
> >> foo_KCONF=CONFIG_FOO
> >>
> >> creating a set of Makefile variables foo_KCONF with the
> >> CONFIG_FOO as values.
> >>
> >> The Makefile then adds this value in the compilation
> >> command with -DKBUILD_KCONF='"CONFIG_FOO"'.
> >
> > Very nice. What would it mean if it lacks this ?
> > This should be explained on the commit log.
> >
> 
> If we lack it then KBUILD_KCONF="" and in /sys the module has the
> kconfig_symbol attribute but with the empty string as content:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$
> 
> I will add it into the commit log.

OK -- I do wonder if instead of an empty string leaving the kconfig_symbol
out is better. If its empty then it can be confusing, and so perhaps better
an "unknown" is better. But skipping the attribute then seems best as then
we can focus on just addressing what it *does mean* when we do have a mapping.
This would allow addressing the semantic gap of modules that lack this and
trying to fix those step by step. To fix those we first need to identify
*why* we can't get an attribute pegged to these - document this perhaps on
kernelnewbies.org/KernelProjects/<pick-a-topic-name-for-your-project>
and then your commit log can reference this for a list of known reasons
and pending work.

> >> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> >> index e7df0f5..8ae9b7f 100644
> >> --- a/scripts/Makefile.lib
> >> +++ b/scripts/Makefile.lib
> >> @@ -89,6 +89,10 @@ multi-objs-m       := $(addprefix $(obj)/,$(multi-objs-m))
> >>  subdir-ym    := $(addprefix $(obj)/,$(subdir-ym))
> >>  obj-dirs     := $(addprefix $(obj)/,$(obj-dirs))
> >>
> >> +# Include Module.ksymb which contains the associations of modules' names
> >> +# and corresponding CONFIG_* options
> >> +include Module.ksymb
> >
> > Is this file always generated? If not prefixing the include call with - would
> > be better. If we are going to make this optional, then an ifdef wrapper would
> > suffice as we know it'd be expected only if the new feature was enabled.
> 
> Yes. This file is always generated. In case a 'git pull' happened
> between two 'make' commands, the associations should be updated. Ok. I
> will add a ifdef.
> 
> >
> >> +
> >>  # These flags are needed for modversions and compiling, so we define them here
> >>  # already
> >>  # $(modname_flags) #defines KBUILD_MODNAME as the name of the module it will
> >> @@ -100,6 +104,9 @@ name-fix = $(squote)$(quote)$(subst $(comma),_,$(subst -,_,$1))$(quote)$(squote)
> >>  basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
> >>  modname_flags  = $(if $(filter 1,$(words $(modname))),\
> >>                   -DKBUILD_MODNAME=$(call name-fix,$(modname)))
> >> +ksym-fix = $(squote)$(quote)$($(subst $(comma),_,$(subst -,_,$1))_KCONF)$(quote)$(squote)
> >> +ksymb_flags = $(if $(filter 1,$(words $(modname))),\
> >> +                 -DKBUILD_KSYMB=$(call ksym-fix, $(modname)))
> >
> > Are clashes possible with this formula? Can we end up with two results for instance?
> > If not what prevents current konfig logic and namespace from a clash ? If we do
> > not have anything to prevent a clash, what can we do to help make such clash not
> > possible ?
> 
> I think the fact that modname is unique prevents from having clashes.
> KBUILD_KSYMB is found according to modname.
> Currently there is no clash because I enforced the Module.ksymb to
> have 1-to-1 mapping.
> The only potential clash I can imagine right now it having the same
> module name but architecture specific CONFIG_* symbol. If there is a
> 1-to-1 mapping in Module.ksymb there should be no clash in the
> makefile.

OK thanks.

  Luis

[toc] | [prev] | [next] | [standalone]

#1464741 — [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

Update modpost to add in *.mod.c files generated for each
module the setting of module attribute kernel_ksymb to
value given by KBUILD_KSYMB macro.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 scripts/mod/modpost.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 48958d3..a105916 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
 	}
 }
 
+static void add_kconfig_symbol(struct buffer *b, struct module *mod)
+{
+	buf_printf(b, "\n");
+	buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
+}
+
 static void write_if_changed(struct buffer *b, const char *fname)
 {
 	char *tmp;
@@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
 		add_depends(&buf, mod, modules);
 		add_moddevtable(&buf, mod);
 		add_srcversion(&buf, mod);
+		add_kconfig_symbol(&buf, mod);
 
 		sprintf(fname, "%s.mod.c", mod->name);
 		write_if_changed(&buf, fname);
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1465884 — Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> Update modpost to add in *.mod.c files generated for each
> module the setting of module attribute kernel_ksymb to
> value given by KBUILD_KSYMB macro.

Please review your patches and update subjects to match what other
types of previous patches look like in terms of format, so
for this patch check changes to scripts/mod/modpost.c and use
similar type of patch prefix.

> 
> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  scripts/mod/modpost.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> index 48958d3..a105916 100644
> --- a/scripts/mod/modpost.c
> +++ b/scripts/mod/modpost.c
> @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
>  	}
>  }
>  
> +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> +{
> +	buf_printf(b, "\n");
> +	buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");

What if its not available? What happens?

> +}
> +
>  static void write_if_changed(struct buffer *b, const char *fname)
>  {
>  	char *tmp;
> @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
>  		add_depends(&buf, mod, modules);
>  		add_moddevtable(&buf, mod);
>  		add_srcversion(&buf, mod);
> +		add_kconfig_symbol(&buf, mod);
>  
>  		sprintf(fname, "%s.mod.c", mod->name);
>  		write_if_changed(&buf, fname);
> -- 
> 2.7.4
> 
> 

-- 
Luis Rodriguez, SUSE LINUX GmbH
Maxfeldstrasse 5; D-90409 Nuernberg

[toc] | [prev] | [next] | [standalone]

#1466819 — Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
>
> On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > Update modpost to add in *.mod.c files generated for each
> > module the setting of module attribute kernel_ksymb to
> > value given by KBUILD_KSYMB macro.
>
> Please review your patches and update subjects to match what other
> types of previous patches look like in terms of format, so
> for this patch check changes to scripts/mod/modpost.c and use
> similar type of patch prefix.
>

OK

>
> >
> > This patch is part of a research project within
> > Google Summer of Code of porting 'make localmodconfig'
> > for backported drivers. The goal is to enable each
> > module to expose in /sys its corresponding CONFIG_* option.
> > The value of this attribute will be dynamically pegged by
> > modpost without requiring extra work from the driver developers.
> > Further, this information will be used by a hardware interogation
> > tool to extract build information about the existing devices.
>
> You can leave this out.
>
> > Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> > ---
> >  scripts/mod/modpost.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > index 48958d3..a105916 100644
> > --- a/scripts/mod/modpost.c
> > +++ b/scripts/mod/modpost.c
> > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> >       }
> >  }
> >
> > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > +{
> > +     buf_printf(b, "\n");
> > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
>
> What if its not available? What happens?
>

If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
More concrete:

prompt:/sys$ cat ./module/mptbase/kconfig_symbol

prompt:/sys$

>
> > +}
> > +
> >  static void write_if_changed(struct buffer *b, const char *fname)
> >  {
> >       char *tmp;
> > @@ -2478,6 +2484,7 @@ int main(int argc, char **argv)
> >               add_depends(&buf, mod, modules);
> >               add_moddevtable(&buf, mod);
> >               add_srcversion(&buf, mod);
> > +             add_kconfig_symbol(&buf, mod);
> >
> >               sprintf(fname, "%s.mod.c", mod->name);
> >               write_if_changed(&buf, fname);
> > --
> > 2.7.4
> >
> >
>
> --
> Luis Rodriguez, SUSE LINUX GmbH
> Maxfeldstrasse 5; D-90409 Nuernberg

[toc] | [prev] | [next] | [standalone]

#1468790 — Re: [RFC PATCH 4/5] Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute

On Sat, Aug 20, 2016 at 05:16:50PM +0200, Cristina-Gabriela Moraru wrote:
> 2016-08-18 20:59 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> >
> > On Wed, Aug 17, 2016 at 09:27:02PM +0200, Cristina Moraru wrote:
> > > diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> > > index 48958d3..a105916 100644
> > > --- a/scripts/mod/modpost.c
> > > +++ b/scripts/mod/modpost.c
> > > @@ -2245,6 +2245,12 @@ static void add_srcversion(struct buffer *b, struct module *mod)
> > >       }
> > >  }
> > >
> > > +static void add_kconfig_symbol(struct buffer *b, struct module *mod)
> > > +{
> > > +     buf_printf(b, "\n");
> > > +     buf_printf(b, "MODULE_INFO(kconfig_symbol, KBUILD_KSYMB);\n");
> >
> > What if its not available? What happens?
> >
> 
> If not available KBUILD_KSYMB is "" and so is set in kconfig_symbol.
> More concrete:
> 
> prompt:/sys$ cat ./module/mptbase/kconfig_symbol
> 
> prompt:/sys$

As I noted in my other e-mail I think its best we just avoid exposing this
then for modules that lack a mapping, and then we make this best effort,
but document the shortcomings very well and super clearly as to why a
backward map is not possible. This then enables us to do work on the build
system to help with this and complete the semantic gap.

  Luis

[toc] | [prev] | [next] | [standalone]

#1464742 — [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile

Trigger the generation of file scripts/Module.ksymb in
Makefile by calling updated scripts/streamline_config.pl
with corresponding parameter (--genmodulesymb). The file
is generated at each compilation considering that
associations may change after tree updates.

This patch is part of a research project within
Google Summer of Code of porting 'make localmodconfig'
for backported drivers. The goal is to enable each
module to expose in /sys its corresponding CONFIG_* option.
The value of this attribute will be dynamically pegged by
modpost without requiring extra work from the driver developers.
Further, this information will be used by a hardware interogation
tool to extract build information about the existing devices.

Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index 393b615..286b949 100644
--- a/Makefile
+++ b/Makefile
@@ -219,6 +219,10 @@ VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
 
 export srctree objtree VPATH
 
+KSYMB_GENERATOR := $(objtree)/scripts/kconfig/streamline_config.pl
+ksymb_gen_command = perl $(KSYMB_GENERATOR) --genmoduleksymb $(objtree) $(Kconfig)
+ksymb_update := $(shell objtree=$(objtree) srctree=$(srctree) $(ksymb_gen_command))
+
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
 # line overrides the setting of ARCH below.  If a native build is happening,
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1465660 — Re: [RFC PATCH 3/5] Trigger Module.ksymb generation in Makefile

On Wed, Aug 17, 2016 at 09:27:01PM +0200, Cristina Moraru wrote:
> Trigger the generation of file scripts/Module.ksymb in
> Makefile by calling updated scripts/streamline_config.pl
> with corresponding parameter (--genmodulesymb). The file
> is generated at each compilation considering that
> associations may change after tree updates.

I really like this approach as you make the resolver
configurable, this lets others experiment and consider
alternatives.

make KSYMB_GENERATOR=/path/research/my-config-generator-is-better

The fact that you can do this should be documented as part of
the patch commit log.

You'll want to wrap this generation under a CONFIG option so only
kernels that want generate this file.

> This patch is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

You can leave this out.

> 
> Signed-off-by: Cristina Moraru <cristina.moraru09@gmail.com>
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 393b615..286b949 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -219,6 +219,10 @@ VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
>  
>  export srctree objtree VPATH
>  
> +KSYMB_GENERATOR := $(objtree)/scripts/kconfig/streamline_config.pl
> +ksymb_gen_command = perl $(KSYMB_GENERATOR) --genmoduleksymb $(objtree) $(Kconfig)

$(PERL)

> +ksymb_update := $(shell objtree=$(objtree) srctree=$(srctree) $(ksymb_gen_command))
> +
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>  # line overrides the setting of ARCH below.  If a native build is happening,
> -- 
> 2.7.4

  Luis

[toc] | [prev] | [next] | [standalone]

#1465889

On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:

> This patchset implements dynamic pegging of kconfig symbol
> into driver modinfo section

First a little bit of motivation here helps, so let me try to
help fill in some gaps. This may help explain what you have
been working on a bit more.

First, for those that were not Cc'd but curious about this work
so far, you can read the patches here:

Original cover letter:

https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com

https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5

There are a few situations in which you may want to extract a
subset of Kconfig CONFIG_* symbols for a system. At least when
only considering modules:

 a) When optimizing build requirements for a kernel for a system.
    That is you boot into a distro kernel and then want to build
    a slim kernel only with sensible kernel configuration options.

 b) When you are on a distribution kernel but the distribution
    kernel provided lacks hardware support for your device, you
    may either want to upgrade the full kernel in which case you
    want to do a) or -- you may want to just a backports release
    which provides just the modules you need, you'd use it on top
    of the distribution kernel.

Are there other *uses* outside of the two above for having a direct
deterministic mapping between module and a kconfig symbol ? What
if we could also do this for built-in code as well ? kconfig-sat [0] is
work that is still being ironed out, but if that was in place and we
handed it a set of required options we want, it should in theory help
resolve the dependencies to give us a functional .config for a kernel.
This of course is a long term possible goal just used as an example
-- but are there other uses?

Another possible gain here is if addressing a deterministic mapping might
help with some of the limitations observed in practice over years with
kconfig and semantics for keconfig. I go into some of this further below.

There are not many options currently available to optimize kernel builds this
way for modules or choose only what kernel kconfig options you need.  One of
the best known efforts is the upstream 'make localmodconfig' added by Steven
years ago to help with a). This has its own set of limitations though. For
backports we also need to address possible kconfig symbol name changes, and the
fact that you may boot a box without a driver loaded and as such not be sure
what new kernel kconfig symbol you may need.

There are shared issues with both goals, addressing shared issues
can ultimately help both. One of the shared known issues is the
lack of a deterministic resolution of a module to a respective
kconfig symbol. Even answering if this is possible or should be
possible requires some work.

The work Cristina has done here for this year's Google Summer of
Code is try to help start looking into addressing the question if
this is possible and then start looking into how we could annotate
such a thing, and later how we'd use it.

[0] https://kernelnewbies.org/KernelProjects/kconfig-sat

> * updates streamline_config.pl to generate the auxiliary file
> scripts/mod/Module.ksymb containing associations of driver file
> names and corresponding kconfig symbols CONFIG_*

Here you make use of existing heuristics -- note however that
heuristics are needed as we have no direct mapping available
nor do we have any rules over how to make such a direct mapping
possible if we wanted this mapping to be deterministic.

> * updates Makefiles to trigger streamline_config.pl call for
> Module.ksymb generation and pass the determined CONFIG_* symbol
> as compilation parameter -D in KBUILD_KSYMB variable

Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
to the build system a config symbol associated with a module, its the right
approach, however re-using streamline_config.pl seems questionable here, and it
seems an optimization might be / should be possible. Likewise we'd need to also
still look into and address *why* some modules don't end up with a respective
kconfig symbol -- and if we wanted to do something about this. Given the stated
goals more importantly would be what the costs are to have a deterministic
mapping.

> * adds kconfig_symb as module attribute

This seems lightweight and follows the logic of the other attributes.
If its too much, this can be a configurable feature.

> * updates modpost to set KBUILD_KSYMB macro as value for
> kbuild_symb attribute

Neat.

> Note: the content of the file Module.ksymb is generated for
> all modules that have only one associate CONFIG option. All
> others are considered to be components linked at the final
> modules but not final modules itselves.

To be clear, you only peg the CONFIG_ option if you are
100% certain of having a direct mapping, is that right ?

> The result of this patchset is the following. All modules from
> /sys expose the correct CONFIG_* symbol in the module attribute
> kconfig_symb with some exceptions. For a total number of 58
> modules, 

What kernel configuration did you use ? What architecture ?
If you use allmodconfig, what do the numbers look like ?

> 4 of them do not:
> 
> snd_seq_midi_event

Takashi, any reason for this to not have a Kconfig symbol ?
If we don't want one, is there perhaps any annotation we
could use to help make a deterministic association if a
respective kconfig symbol is needed ?

> mptscsih

Same thing -- for both I suppose we can infer that these
are just helpers, and can / should be enabled if any of
the devices are found to be present / required.

I can imagine that in certain situations a helper like
the above might really just be optional rather than required,
but I'd hope such an optional nature should be reflected via
Kconfig, the default setting being the recommended option to
assume as safe then, unless input is provided to help fill
in the gap and disable / force enable certain options.

> libahci

In this case libahci is spinkled over a lot of lines on the Makefile,
its a module though, and there is no direct association of dependency
relationship other than the Makefile association. Using a kconfig
symbol would replace the Makefile clutter but make the assocation
and requirement of the dependency explicit through Kconfig. Is
that desirable or should we just make the assumption that in these
cases we could assume one Kconfig symbol that adds a module implicitly
is equivalent as if the module had a kconfig symbol and we had done
"select foo" for each ? Note though that select is heavy handed [1] --
and its over use can lead to issues. For instance its known that
select does not check for dependencies, its a heavy handed way of
enabling options needed by architectures. So using select here may
not be desirable. Using "depends on" here would work but it can hide
from the menu drivers until you have enabled its dependency.

[1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e

> mptbase

Same situation as  mptscsih.

> After a short research:
> 
> For mptscsih - ./drivers/message/fusion/Makefile
> obj-$(CONFIG_FUSION_SPI)	+= mptbase.o mptscsih.o mptspi.o
> obj-$(CONFIG_FUSION_FC)		+= mptbase.o mptscsih.o mptfc.o
> obj-$(CONFIG_FUSION_SAS)	+= mptbase.o mptscsih.o mptsas.o
> As appears in the Makefile mptscsi is part of more config
> options so it's excluded because it's not considered a module
> by itself but, somehow gets compiled as module. Same for the
> mptbase. I still have to understand better the Kbuild logic so
> I cannot state that this is loose practice.

Another reason for this loose use could also be that using "select"
in Kconfig is heavy handed and can lead to recursive issues, refer
to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
dependency limitation / resolution") for more information and
a clear example of the issue. There is also a list of commits
with possible resolutions to issues, so that can be inspected further
to try to understand if perhaps some of the lack of kconfig symbols
and respective deterministic mapping from module to kconfig is due
to prior issues with select. I doubt it given that otherwise we'd
have at least symbols for these -- but worth taking into consideration
*if* we do then want to add symbols for them, we do not want to lead
to recursive issues if we add these new symbols.

If we add a kconfig symbol for modules lacking them, what is a
strategical way to do so in such a way we avoid known past
issues but also paves the way for the future ?

> For libahci there are multiple CONFIG_ symbols for different
> architectures.
> 
> Although past discussion recommended not to use streamline_config,
> this patchset still relies on it, for the sake of the proof of
> concept. 

OK.

> Although the result is close to the target it still does
> not provide complete correctness. However it can be replaced by
> creating another script

Or C code.

> which tries to determine better the
> module <-> CONFIG associations and output them in auxiliary file
> Module.ksymb. Maybe this way we could also determine all CONFIGs
> for a particular driver, not only the exact one that enables it.
> 
> The auxiliary file is necessary because the Makefile itself does
> not have a mapping. 

Clarification: at least not an easy O(1) quick mapping.

> The makefile includes the config file with
> directive include which creates a series of internal variables
> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
> other Makefiles, lines like 
> 
> obj-$(CONFIG_FOO) = foo.o 
> 
> are resolved in place to obj-y, obj-m etc according to 'make'
> logic and the association is lost.
> 
> Finally, what does this patchset provide is an infrastructure
> to dinamically peg CONFIG_* options to associate drivers using
> the mapping from Module.ksymb file. Generation of Module.ksymb
> can be replaced but keeping the same format permit the usage of
> the other patches.

OK this makes sense, so as it stands it seems we need a better
way to generate the Module.ksymb and then also address those
modules which lack a current deterministic mapping, and address
a resolution with the community on that.

> This patchset is part of a research project within
> Google Summer of Code of porting 'make localmodconfig'
> for backported drivers. The goal is to enable each
> module to expose in /sys its corresponding CONFIG_* option.
> The value of this attribute will be dynamically pegged by
> modpost without requiring extra work from the driver developers.
> Further, this information will be used by a hardware interogation
> tool to extract build information about the existing devices.

Good stuff!

  Luis

> Cristina Moraru (5):
>   Add generation of Module.symb in streamline_config
>   Add CONFIG symbol to module as compilation parameter
>   Trigger Module.ksymb generation in Makefile
>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>   Add kconf_symb as kernel module attribute
> 
>  Makefile                             |  4 ++++
>  include/linux/module.h               |  1 +
>  kernel/module.c                      |  2 ++
>  scripts/Makefile.lib                 |  9 ++++++++-
>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>  scripts/mod/modpost.c                |  7 +++++++
>  6 files changed, 51 insertions(+), 2 deletions(-)
> 
> -- 
> 2.7.4
> 
> 

[toc] | [prev] | [next] | [standalone]

#1466243

On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> 
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
> 
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
> 
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
> 
> Original cover letter:
> 
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> 
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> 
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
> 
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
> 
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.

c) Having the mapping in sysfs would allow to simplify
streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
did not depend on streamline_config.pl :). One idea would be to generate
the Module.ksymb in a similar way we generate the modules.builtin file:
Generate an alternate include/config/*.conf with all CONFIG_FOO=m
replaced with

CONFIG_FOO=m-CONFIG_FOO

and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
create the mapping. This would also properly cover cases where we build
the $(obj-m) list from another list. It would certainly create other
corner cases, but it's worth trying IMO.

Another thing is that we do not necessarily need to record this
information in .modinfo, but we can generate a list in
/lib/modules/`uname -r`/ for consumption. It could also include drivers
that are builtin in the current configuration.

Michal

[toc] | [prev] | [next] | [standalone]

#1468009

2016-08-19 11:07 GMT+02:00 Michal Marek <mmarek@suse.com>:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>
>>> This patchset implements dynamic pegging of kconfig symbol
>>> into driver modinfo section
>>
>> First a little bit of motivation here helps, so let me try to
>> help fill in some gaps. This may help explain what you have
>> been working on a bit more.
>>
>> First, for those that were not Cc'd but curious about this work
>> so far, you can read the patches here:
>>
>> Original cover letter:
>>
>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>
>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>
>> There are a few situations in which you may want to extract a
>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> only considering modules:
>>
>>  a) When optimizing build requirements for a kernel for a system.
>>     That is you boot into a distro kernel and then want to build
>>     a slim kernel only with sensible kernel configuration options.
>>
>>  b) When you are on a distribution kernel but the distribution
>>     kernel provided lacks hardware support for your device, you
>>     may either want to upgrade the full kernel in which case you
>>     want to do a) or -- you may want to just a backports release
>>     which provides just the modules you need, you'd use it on top
>>     of the distribution kernel.
>
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl. Only if the patch
> did not depend on streamline_config.pl :). One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
>
> CONFIG_FOO=m-CONFIG_FOO
>
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.
>

That's interesting.

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.
>
> Michal

Thank you,

Cristina

[toc] | [prev] | [next] | [standalone]

#1468893

On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> > On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> > 
> >> This patchset implements dynamic pegging of kconfig symbol
> >> into driver modinfo section
> > 
> > First a little bit of motivation here helps, so let me try to
> > help fill in some gaps. This may help explain what you have
> > been working on a bit more.
> > 
> > First, for those that were not Cc'd but curious about this work
> > so far, you can read the patches here:
> > 
> > Original cover letter:
> > 
> > https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> > 
> > https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> > https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> > https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> > https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> > https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> > 
> > There are a few situations in which you may want to extract a
> > subset of Kconfig CONFIG_* symbols for a system. At least when
> > only considering modules:
> > 
> >  a) When optimizing build requirements for a kernel for a system.
> >     That is you boot into a distro kernel and then want to build
> >     a slim kernel only with sensible kernel configuration options.
> > 
> >  b) When you are on a distribution kernel but the distribution
> >     kernel provided lacks hardware support for your device, you
> >     may either want to upgrade the full kernel in which case you
> >     want to do a) or -- you may want to just a backports release
> >     which provides just the modules you need, you'd use it on top
> >     of the distribution kernel.
> 
> c) Having the mapping in sysfs would allow to simplify
> streamline_config.pl avoid parsing Makefiles in perl.

Indeed, this was actually a side possible goal here :) only it seems
we totally forgot about it while addressing the work.

> Only if the patch did not depend on streamline_config.pl :)

Indeed, lets get rid of that dependency.

> One idea would be to generate
> the Module.ksymb in a similar way we generate the modules.builtin file:
> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> replaced with
> 
> CONFIG_FOO=m-CONFIG_FOO
> 
> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
> create the mapping. This would also properly cover cases where we build
> the $(obj-m) list from another list. It would certainly create other
> corner cases, but it's worth trying IMO.

Neat, would that cover lib-m then as well or other targets that would generate 
modules ?

> Another thing is that we do not necessarily need to record this
> information in .modinfo, but we can generate a list in
> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> that are builtin in the current configuration.

Well so one future use would be for instance a 'make udevconfig' which
in theory should be able then to scrape all possible related needed
device CONFIG_'s and then give you an optimal kernel configuration.
It'd be nice to be able to extract this from the kernel and modules
without needing build sources in /lib/modules/`uname -r`/.

Thoughts?

  Luis

[toc] | [prev] | [next] | [standalone]

#1469343

On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>>>
>>>> This patchset implements dynamic pegging of kconfig symbol
>>>> into driver modinfo section
>>>
>>> First a little bit of motivation here helps, so let me try to
>>> help fill in some gaps. This may help explain what you have
>>> been working on a bit more.
>>>
>>> First, for those that were not Cc'd but curious about this work
>>> so far, you can read the patches here:
>>>
>>> Original cover letter:
>>>
>>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>>>
>>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>>>
>>> There are a few situations in which you may want to extract a
>>> subset of Kconfig CONFIG_* symbols for a system. At least when
>>> only considering modules:
>>>
>>>  a) When optimizing build requirements for a kernel for a system.
>>>     That is you boot into a distro kernel and then want to build
>>>     a slim kernel only with sensible kernel configuration options.
>>>
>>>  b) When you are on a distribution kernel but the distribution
>>>     kernel provided lacks hardware support for your device, you
>>>     may either want to upgrade the full kernel in which case you
>>>     want to do a) or -- you may want to just a backports release
>>>     which provides just the modules you need, you'd use it on top
>>>     of the distribution kernel.
>>
>> c) Having the mapping in sysfs would allow to simplify
>> streamline_config.pl avoid parsing Makefiles in perl.
> 
> Indeed, this was actually a side possible goal here :) only it seems
> we totally forgot about it while addressing the work.
> 
>> Only if the patch did not depend on streamline_config.pl :)
> 
> Indeed, lets get rid of that dependency.
> 
>> One idea would be to generate
>> the Module.ksymb in a similar way we generate the modules.builtin file:
>> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> replaced with
>>
>> CONFIG_FOO=m-CONFIG_FOO
>>
>> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
                                       should be obj-m-CONFIG_%

>> create the mapping. This would also properly cover cases where we build
>> the $(obj-m) list from another list. It would certainly create other
>> corner cases, but it's worth trying IMO.
> 
> Neat, would that cover lib-m then as well or other targets that would generate 
> modules ?

This would have to be handled as well by the patch. Obviously it's not
going to be as simple as described in a single paragraph of text :).


>> Another thing is that we do not necessarily need to record this
>> information in .modinfo, but we can generate a list in
>> /lib/modules/`uname -r`/ for consumption. It could also include drivers
>> that are builtin in the current configuration.
> 
> Well so one future use would be for instance a 'make udevconfig' which
> in theory should be able then to scrape all possible related needed
> device CONFIG_'s and then give you an optimal kernel configuration.
> It'd be nice to be able to extract this from the kernel and modules
> without needing build sources in /lib/modules/`uname -r`/.

I meant that the file can be installed by modules_install. Like you have
/lib/modules/`uname -r`/modules.builtin with recent kernels and recent
distro packages. But really, it's a minor issue how the information is
presented to userspace.

Michal

[toc] | [prev] | [next] | [standalone]

#1469569

On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
> >>>
> >>>> This patchset implements dynamic pegging of kconfig symbol
> >>>> into driver modinfo section
> >>>
> >>> First a little bit of motivation here helps, so let me try to
> >>> help fill in some gaps. This may help explain what you have
> >>> been working on a bit more.
> >>>
> >>> First, for those that were not Cc'd but curious about this work
> >>> so far, you can read the patches here:
> >>>
> >>> Original cover letter:
> >>>
> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
> >>>
> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
> >>>
> >>> There are a few situations in which you may want to extract a
> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
> >>> only considering modules:
> >>>
> >>>  a) When optimizing build requirements for a kernel for a system.
> >>>     That is you boot into a distro kernel and then want to build
> >>>     a slim kernel only with sensible kernel configuration options.
> >>>
> >>>  b) When you are on a distribution kernel but the distribution
> >>>     kernel provided lacks hardware support for your device, you
> >>>     may either want to upgrade the full kernel in which case you
> >>>     want to do a) or -- you may want to just a backports release
> >>>     which provides just the modules you need, you'd use it on top
> >>>     of the distribution kernel.
> >>
> >> c) Having the mapping in sysfs would allow to simplify
> >> streamline_config.pl avoid parsing Makefiles in perl.
> > 
> > Indeed, this was actually a side possible goal here :) only it seems
> > we totally forgot about it while addressing the work.
> > 
> >> Only if the patch did not depend on streamline_config.pl :)
> > 
> > Indeed, lets get rid of that dependency.
> > 
> >> One idea would be to generate
> >> the Module.ksymb in a similar way we generate the modules.builtin file:
> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
> >> replaced with
> >>
> >> CONFIG_FOO=m-CONFIG_FOO
> >>
> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES)) to
>                                        should be obj-m-CONFIG_%
> 
> >> create the mapping. This would also properly cover cases where we build
> >> the $(obj-m) list from another list. It would certainly create other
> >> corner cases, but it's worth trying IMO.
> > 
> > Neat, would that cover lib-m then as well or other targets that would generate 
> > modules ?
> 
> This would have to be handled as well by the patch. Obviously it's not
> going to be as simple as described in a single paragraph of text :).

Ah OK, it was not clear to me if your suggestion captured all modules already.
There may still be a way. We just have to look harder. Then what precise
target was used would be irrelevant.

> >> Another thing is that we do not necessarily need to record this
> >> information in .modinfo, but we can generate a list in
> >> /lib/modules/`uname -r`/ for consumption. It could also include drivers
> >> that are builtin in the current configuration.
> > 
> > Well so one future use would be for instance a 'make udevconfig' which
> > in theory should be able then to scrape all possible related needed
> > device CONFIG_'s and then give you an optimal kernel configuration.
> > It'd be nice to be able to extract this from the kernel and modules
> > without needing build sources in /lib/modules/`uname -r`/.
> 
> I meant that the file can be installed by modules_install. Like you have
> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
> distro packages.

Got it.

> But really, it's a minor issue how the information is
> presented to userspace.

To us perhaps, but to userspace and users this is a big deal. It'd be nice
to be able to construct a kernel config only with what CONFIG's your hardware
needs, doing this should mean trying to expose enough information in a way
userspace is easily able to handle. The reason to go with the mod section stuff
was this was already being picked up by udevadm for its own hardware db info,
so this seemed like a logical evolution, and it would not require your
kernel sources installed.

  Luis

[toc] | [prev] | [next] | [standalone]

#1469598

unsubscribe backports

On 8/24/16, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> On Wed, Aug 24, 2016 at 01:05:04PM +0200, Michal Marek wrote:
>> On 2016-08-23 23:32, Luis R. Rodriguez wrote:
>> > On Fri, Aug 19, 2016 at 11:07:36AM +0200, Michal Marek wrote:
>> >> On 2016-08-18 19:55, Luis R. Rodriguez wrote:
>> >>> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>> >>>
>> >>>> This patchset implements dynamic pegging of kconfig symbol
>> >>>> into driver modinfo section
>> >>>
>> >>> First a little bit of motivation here helps, so let me try to
>> >>> help fill in some gaps. This may help explain what you have
>> >>> been working on a bit more.
>> >>>
>> >>> First, for those that were not Cc'd but curious about this work
>> >>> so far, you can read the patches here:
>> >>>
>> >>> Original cover letter:
>> >>>
>> >>> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>> >>>
>> >>> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
>> >>> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
>> >>> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
>> >>> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
>> >>> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>> >>>
>> >>> There are a few situations in which you may want to extract a
>> >>> subset of Kconfig CONFIG_* symbols for a system. At least when
>> >>> only considering modules:
>> >>>
>> >>>  a) When optimizing build requirements for a kernel for a system.
>> >>>     That is you boot into a distro kernel and then want to build
>> >>>     a slim kernel only with sensible kernel configuration options.
>> >>>
>> >>>  b) When you are on a distribution kernel but the distribution
>> >>>     kernel provided lacks hardware support for your device, you
>> >>>     may either want to upgrade the full kernel in which case you
>> >>>     want to do a) or -- you may want to just a backports release
>> >>>     which provides just the modules you need, you'd use it on top
>> >>>     of the distribution kernel.
>> >>
>> >> c) Having the mapping in sysfs would allow to simplify
>> >> streamline_config.pl avoid parsing Makefiles in perl.
>> >
>> > Indeed, this was actually a side possible goal here :) only it seems
>> > we totally forgot about it while addressing the work.
>> >
>> >> Only if the patch did not depend on streamline_config.pl :)
>> >
>> > Indeed, lets get rid of that dependency.
>> >
>> >> One idea would be to generate
>> >> the Module.ksymb in a similar way we generate the modules.builtin
>> >> file:
>> >> Generate an alternate include/config/*.conf with all CONFIG_FOO=m
>> >> replaced with
>> >>
>> >> CONFIG_FOO=m-CONFIG_FOO
>> >>
>> >> and in the Makefile, iterate over $(filter m-CONFIG_%, $(.VARIABLES))
>> >> to
>>                                        should be obj-m-CONFIG_%
>>
>> >> create the mapping. This would also properly cover cases where we
>> >> build
>> >> the $(obj-m) list from another list. It would certainly create other
>> >> corner cases, but it's worth trying IMO.
>> >
>> > Neat, would that cover lib-m then as well or other targets that would
>> > generate
>> > modules ?
>>
>> This would have to be handled as well by the patch. Obviously it's not
>> going to be as simple as described in a single paragraph of text :).
>
> Ah OK, it was not clear to me if your suggestion captured all modules
> already.
> There may still be a way. We just have to look harder. Then what precise
> target was used would be irrelevant.
>
>> >> Another thing is that we do not necessarily need to record this
>> >> information in .modinfo, but we can generate a list in
>> >> /lib/modules/`uname -r`/ for consumption. It could also include
>> >> drivers
>> >> that are builtin in the current configuration.
>> >
>> > Well so one future use would be for instance a 'make udevconfig' which
>> > in theory should be able then to scrape all possible related needed
>> > device CONFIG_'s and then give you an optimal kernel configuration.
>> > It'd be nice to be able to extract this from the kernel and modules
>> > without needing build sources in /lib/modules/`uname -r`/.
>>
>> I meant that the file can be installed by modules_install. Like you have
>> /lib/modules/`uname -r`/modules.builtin with recent kernels and recent
>> distro packages.
>
> Got it.
>
>> But really, it's a minor issue how the information is
>> presented to userspace.
>
> To us perhaps, but to userspace and users this is a big deal. It'd be nice
> to be able to construct a kernel config only with what CONFIG's your
> hardware
> needs, doing this should mean trying to expose enough information in a way
> userspace is easily able to handle. The reason to go with the mod section
> stuff
> was this was already being picked up by udevadm for its own hardware db
> info,
> so this seemed like a logical evolution, and it would not require your
> kernel sources installed.
>
>   Luis
> --
> To unsubscribe from this list: send the line "unsubscribe backports" in
>

[toc] | [prev] | [next] | [standalone]

#1468000

2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@kernel.org>:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
>
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
>
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
>
> Original cover letter:
>
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@gmail.com
>
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
>
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
>
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.
>
> Are there other *uses* outside of the two above for having a direct
> deterministic mapping between module and a kconfig symbol ? What
> if we could also do this for built-in code as well ? kconfig-sat [0] is
> work that is still being ironed out, but if that was in place and we
> handed it a set of required options we want, it should in theory help
> resolve the dependencies to give us a functional .config for a kernel.
> This of course is a long term possible goal just used as an example
> -- but are there other uses?
>
I can't think of any examples. I will add this ones in the cover
letter as motivation.

> Another possible gain here is if addressing a deterministic mapping might
> help with some of the limitations observed in practice over years with
> kconfig and semantics for keconfig. I go into some of this further below.
>
> There are not many options currently available to optimize kernel builds this
> way for modules or choose only what kernel kconfig options you need.  One of
> the best known efforts is the upstream 'make localmodconfig' added by Steven
> years ago to help with a). This has its own set of limitations though. For
> backports we also need to address possible kconfig symbol name changes, and the
> fact that you may boot a box without a driver loaded and as such not be sure
> what new kernel kconfig symbol you may need.
>
> There are shared issues with both goals, addressing shared issues
> can ultimately help both. One of the shared known issues is the
> lack of a deterministic resolution of a module to a respective
> kconfig symbol. Even answering if this is possible or should be
> possible requires some work.
>
> The work Cristina has done here for this year's Google Summer of
> Code is try to help start looking into addressing the question if
> this is possible and then start looking into how we could annotate
> such a thing, and later how we'd use it.
>
> [0] https://kernelnewbies.org/KernelProjects/kconfig-sat
>
>> * updates streamline_config.pl to generate the auxiliary file
>> scripts/mod/Module.ksymb containing associations of driver file
>> names and corresponding kconfig symbols CONFIG_*
>
> Here you make use of existing heuristics -- note however that
> heuristics are needed as we have no direct mapping available
> nor do we have any rules over how to make such a direct mapping
> possible if we wanted this mapping to be deterministic.
>
>> * updates Makefiles to trigger streamline_config.pl call for
>> Module.ksymb generation and pass the determined CONFIG_* symbol
>> as compilation parameter -D in KBUILD_KSYMB variable
>
> Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
> to the build system a config symbol associated with a module, its the right
> approach, however re-using streamline_config.pl seems questionable here, and it
> seems an optimization might be / should be possible. Likewise we'd need to also
> still look into and address *why* some modules don't end up with a respective
> kconfig symbol -- and if we wanted to do something about this. Given the stated
> goals more importantly would be what the costs are to have a deterministic
> mapping.
>

Yes. I've noticed. I will change it completely.

>> * adds kconfig_symb as module attribute
>
> This seems lightweight and follows the logic of the other attributes.
> If its too much, this can be a configurable feature.
>
>> * updates modpost to set KBUILD_KSYMB macro as value for
>> kbuild_symb attribute
>
> Neat.
>
>> Note: the content of the file Module.ksymb is generated for
>> all modules that have only one associate CONFIG option. All
>> others are considered to be components linked at the final
>> modules but not final modules itselves.
>
> To be clear, you only peg the CONFIG_ option if you are
> 100% certain of having a direct mapping, is that right ?
>

Yes. And there are some drivers which have duplicate values in the hash.
foo.o -> CONFIG_FOO CONFIG_FOO

Although this one has one match it's filtered as having 2 CONFIGs.

>> The result of this patchset is the following. All modules from
>> /sys expose the correct CONFIG_* symbol in the module attribute
>> kconfig_symb with some exceptions. For a total number of 58
>> modules,
>
> What kernel configuration did you use ? What architecture ?
> If you use allmodconfig, what do the numbers look like ?
>

I have a virtual machine in which I created the configuration with
make localmodconfig and installed the new kernel.
$  uname -a
Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
x86_64 x86_64 GNU/Linux

Should I send you the .config file ?

>> 4 of them do not:
>>
>> snd_seq_midi_event
>
> Takashi, any reason for this to not have a Kconfig symbol ?
> If we don't want one, is there perhaps any annotation we
> could use to help make a deterministic association if a
> respective kconfig symbol is needed ?
>
>> mptscsih
>
> Same thing -- for both I suppose we can infer that these
> are just helpers, and can / should be enabled if any of
> the devices are found to be present / required.
>
> I can imagine that in certain situations a helper like
> the above might really just be optional rather than required,
> but I'd hope such an optional nature should be reflected via
> Kconfig, the default setting being the recommended option to
> assume as safe then, unless input is provided to help fill
> in the gap and disable / force enable certain options.
>
>> libahci
>
> In this case libahci is spinkled over a lot of lines on the Makefile,
> its a module though, and there is no direct association of dependency
> relationship other than the Makefile association. Using a kconfig
> symbol would replace the Makefile clutter but make the assocation
> and requirement of the dependency explicit through Kconfig. Is
> that desirable or should we just make the assumption that in these
> cases we could assume one Kconfig symbol that adds a module implicitly
> is equivalent as if the module had a kconfig symbol and we had done
> "select foo" for each ? Note though that select is heavy handed [1] --
> and its over use can lead to issues. For instance its known that
> select does not check for dependencies, its a heavy handed way of
> enabling options needed by architectures. So using select here may
> not be desirable. Using "depends on" here would work but it can hide
> from the menu drivers until you have enabled its dependency.
>
> [1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e
>
>> mptbase
>
> Same situation as  mptscsih.
>
>> After a short research:
>>
>> For mptscsih - ./drivers/message/fusion/Makefile
>> obj-$(CONFIG_FUSION_SPI)      += mptbase.o mptscsih.o mptspi.o
>> obj-$(CONFIG_FUSION_FC)               += mptbase.o mptscsih.o mptfc.o
>> obj-$(CONFIG_FUSION_SAS)      += mptbase.o mptscsih.o mptsas.o
>> As appears in the Makefile mptscsi is part of more config
>> options so it's excluded because it's not considered a module
>> by itself but, somehow gets compiled as module. Same for the
>> mptbase. I still have to understand better the Kbuild logic so
>> I cannot state that this is loose practice.
>
> Another reason for this loose use could also be that using "select"
> in Kconfig is heavy handed and can lead to recursive issues, refer
> to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
> dependency limitation / resolution") for more information and
> a clear example of the issue. There is also a list of commits
> with possible resolutions to issues, so that can be inspected further
> to try to understand if perhaps some of the lack of kconfig symbols
> and respective deterministic mapping from module to kconfig is due
> to prior issues with select. I doubt it given that otherwise we'd
> have at least symbols for these -- but worth taking into consideration
> *if* we do then want to add symbols for them, we do not want to lead
> to recursive issues if we add these new symbols.
>
> If we add a kconfig symbol for modules lacking them, what is a
> strategical way to do so in such a way we avoid known past
> issues but also paves the way for the future ?
>
>> For libahci there are multiple CONFIG_ symbols for different
>> architectures.
>>
>> Although past discussion recommended not to use streamline_config,
>> this patchset still relies on it, for the sake of the proof of
>> concept.
>
> OK.
>
>> Although the result is close to the target it still does
>> not provide complete correctness. However it can be replaced by
>> creating another script
>
> Or C code.
>
>> which tries to determine better the
>> module <-> CONFIG associations and output them in auxiliary file
>> Module.ksymb. Maybe this way we could also determine all CONFIGs
>> for a particular driver, not only the exact one that enables it.
>>
>> The auxiliary file is necessary because the Makefile itself does
>> not have a mapping.
>
> Clarification: at least not an easy O(1) quick mapping.
>
>> The makefile includes the config file with
>> directive include which creates a series of internal variables
>> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
>> other Makefiles, lines like
>>
>> obj-$(CONFIG_FOO) = foo.o
>>
>> are resolved in place to obj-y, obj-m etc according to 'make'
>> logic and the association is lost.
>>
>> Finally, what does this patchset provide is an infrastructure
>> to dinamically peg CONFIG_* options to associate drivers using
>> the mapping from Module.ksymb file. Generation of Module.ksymb
>> can be replaced but keeping the same format permit the usage of
>> the other patches.
>
> OK this makes sense, so as it stands it seems we need a better
> way to generate the Module.ksymb and then also address those
> modules which lack a current deterministic mapping, and address
> a resolution with the community on that.
>
>> This patchset is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> Good stuff!
>
>   Luis
>
>> Cristina Moraru (5):
>>   Add generation of Module.symb in streamline_config
>>   Add CONFIG symbol to module as compilation parameter
>>   Trigger Module.ksymb generation in Makefile
>>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>>   Add kconf_symb as kernel module attribute
>>
>>  Makefile                             |  4 ++++
>>  include/linux/module.h               |  1 +
>>  kernel/module.c                      |  2 ++
>>  scripts/Makefile.lib                 |  9 ++++++++-
>>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>>  scripts/mod/modpost.c                |  7 +++++++
>>  6 files changed, 51 insertions(+), 2 deletions(-)
>>
>> --
>> 2.7.4
>>
>>

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web