Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1407760 > unrolled thread

Re: [RFT v2] iommu/amd: use subsys_initcall() on amdv2 iommu

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFT v2] iommu/amd: use subsys_initcall() on amdv2 iommu "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-05-27 02:50 +0200
    [RFT v3] drm: use late_initcall() for amdkfd and radeon "Luis R. Rodriguez" <mcgrof@kernel.org> - 2016-05-27 03:20 +0200
      Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon Oded Gabbay <oded.gabbay@gmail.com> - 2016-05-29 17:00 +0200
      Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon Daniel Vetter <daniel@ffwll.ch> - 2016-05-29 20:30 +0200

#1407760 — Re: [RFT v2] iommu/amd: use subsys_initcall() on amdv2 iommu

On Mon, Apr 25, 2016 at 12:23:51PM +0200, Joerg Roedel wrote:
> On Mon, Apr 18, 2016 at 02:03:50PM +0200, Luis R. Rodriguez wrote:
> > You said that with my patch you saw AMD IOMMUv2 kick off first,
> > that was intentional as I thought that's what you needed. Can
> > someone please describe the requirements?
> > 
> > Also what does drm use that you say has a conflict already? What
> > drm code are we talking about exactly ?
> 
> The required order is:
> 
> 	1. AMD IOMMUv1 (in-kernel only, initialized at boot time after PCI)
> 	2. AMD IOMMUv2 (in-kernel or as module, provides demand paging
> 			and uses v1 interfaces to talk to the IOMMU)
> 	3. AMD-KFD     (Implements compute offloading to the GPU and
> 			uses AMD IOMMUv2 functionality, also provides a
> 			symbol for the radeon driver)
> 	4. DRM with    (Checks if the symbol provided by AMD-KFD is
> 	   Radeon	available at init time and does the KFD
> 			initialization from there, because the GPU needs
> 			to be up first)
> 
> So AMD IOMMUv2 does not initialize (but it does load to have the symbols
> available for drivers that optionally use its functionality) without the
> AMD IOMMUv1 driver, as it can't access the IOMMU hardware then.
> 
> AMD-KFD does not load without AMD IOMMUv2 being loaded, as it depends on
> its symbols. AMD-KFD on the other hand needs to be loaded before the
> radeon driver (but this it not enforced by symbols), because otherwise
> the radeon driver will not initialize the AMD-KFD driver.
> 
> When AMD-KFD is loaded and you load radeon then, you get the KFD
> functionality in the kernel. Then you can move on to the fun getting the
> userspace running and actually execute anything on the GPU. But thats
> another story.
> 
> I know what you think, and I agree: It's a big mess :)

Its no concern for me that its a mess, frankly most drivers are. This however
illustrates a semantic gap in the driver core when you have more than 3
dependencies after a rootfs_initcall(), given that the buck stops at
late_initcall(), only 3 levels above. With 4 dependencies you run out
of semantics to specify explicit requirements.

Ties can be disputed through link order though, but this is obviously fragile,
and the dependencies are then left implicit. Nothing vets for correctness,
either in software or through static analysers.

To summarize the specific code in question is (and order required):

AMD IOMMUv1:    arch/x86/kernel/pci-dma.c               rootfs_initcall(pci_iommu_init);
AMD IOMMUv2:    drivers/iommu/amd_iommu_v2.c            module_init(amd_iommu_v2_init); (device_initcall())
AMD KFD:        drivers/gpu/drm/amd/amdkfd/kfd_module.c module_init(kfd_module_init); (device_initcall())
AMD Radeon:     drivers/gpu/drm/radeon/radeon_drv.c     module_init(radeon_init); (device_initcall())

Commit 1bacc894c227fad8a7 ("drivers: Move iommu/ before gpu/ in Makefile") resolved
the race between IOMMU and GPU drivers this way, however an alternative is to make
the dependencies in levels explicit by making the amdkfd and radeon drivers both
use late_initcall(), this however is technically still racy specially since you
note that the amdkfd driver is not loaded prior to radeon through symbol
dependencies, if happens to be loaded it then you get KFD functionality,
otherwise you don't.

Making both amdkfd and radeon use late_initcall() would actually work now though
but that's only because the link order also happens to match the dependency
requirements:

drivers/gpu/drm/Makefile
...
obj-$(CONFIG_HSA_AMD) += amd/amdkfd/                                            
obj-$(CONFIG_DRM_RADEON)+= radeon/  
...

Since we currently lack proper semantics to define clear dependencies for all
this reverting 1bacc894c227fad8a7 after using late_initcall() on both amdkfd
and radeon would not be useful other than to just test the race fix, given that
such work around would still be present also on drivers/gpu/drm/Makefile to
account for the race between amdkfd and radeon. Reverting 1bacc894c227fad8a7
after using late_initcall() on both amdkfd and radeon would just bump the
race work around another level.

Modifying both amdkfd and radeon to use late_initcall() however seems well
justified for now, and given the current state of affairs with link order
one should be able to then correctly build all these things as built-in
and it should work correctly.

I'm still not satisfied with the issues on semantics here though. A fix
for that, if desired, should be possible using linker-tables, currently
in RFCv2 [0] stage. Linker tables offer practically infinite number (as
long as a valid ELF section name, as the order level is part of the
section name) of order levels. It also would offer the opportunity
to build dependencies outside of the driver core layer, so you can
customize the dependency map per subsystem as you see fit.

So -- for now I'll submit a patch to just use late_initcall(), in the
future we can evaluate linker table uses to make these dependencies
explicit. This is perhaps a good test case to illustrate limitations
of the current driver model.

[0] http://lkml.kernel.org/r/1455889559-9428-1-git-send-email-mcgrof@kernel.org

  Luis

[toc] | [next] | [standalone]

#1407765 — [RFT v3] drm: use late_initcall() for amdkfd and radeon

To get KFD support in radeon we need the following
initialization to happen in this order, their
respective driver file that has its init routine
listed next to it:

0. AMD IOMMUv1:    arch/x86/kernel/pci-dma.c
1. AMD IOMMUv2:    drivers/iommu/amd_iommu_v2.c
2. AMD KFD:        drivers/gpu/drm/amd/amdkfd/kfd_module.c
3. AMD Radeon:     drivers/gpu/drm/radeon/radeon_drv.c

Order is rather implicit, but these drivers can currently
only do so much given the amount of leg room available.
Below are the respective init routines and how they are
initialized:

arch/x86/kernel/pci-dma.c               rootfs_initcall(pci_iommu_init);
drivers/iommu/amd_iommu_v2.c            module_init(amd_iommu_v2_init);
drivers/gpu/drm/amd/amdkfd/kfd_module.c module_init(kfd_module_init);
drivers/gpu/drm/radeon/radeon_drv.c     module_init(radeon_init);

When a driver is built-in module_init() folds to use
device_initcall(), and we have the following possible
orders:

	#define pure_initcall(fn)    __define_initcall(fn, 0)
	#define core_initcall(fn)    __define_initcall(fn, 1)
	#define postcore_initcall(fn)__define_initcall(fn, 2)
	#define arch_initcall(fn)    __define_initcall(fn, 3)
	#define subsys_initcall(fn)  __define_initcall(fn, 4)
	#define fs_initcall(fn)      __define_initcall(fn, 5)
        ---------------------------------------------------------
	#define rootfs_initcall(fn)  __define_initcall(fn, rootfs)
	#define device_initcall(fn)  __define_initcall(fn, 6)
	#define late_initcall(fn)    __define_initcall(fn, 7)

Since we start off from rootfs_initcall(), it gives us 3 more
levels of leg room to play with for order semantics, this isn't
enough to address all required levels of dependencies, this
is specially true given that AMD-KFD needs to be loaded before
the radeon driver -- -but this it not enforced by symbols.
If the AMD-KFD driver is not loaded prior to the radeon driver
because otherwise the radeon driver will not initialize the
AMD-KFD driver and you get no KFD functionality in userspace.

Commit 1bacc894c227fad8a7 ("drivers: Move iommu/ before gpu/ in
Makefile") works around some of the possibe races between
the AMD IOMMU v2 and GPU drivers by changing the link order.
This is fragile, however its the bets we can do, given that
making the GPU drivers use late_initcall() would also implicate
a similar race between them. That possible race is fortunatley
addressed given that the drm Makefile currently has amdkfd
linked prior to radeon:

drivers/gpu/drm/Makefile
...
obj-$(CONFIG_HSA_AMD) += amd/amdkfd/
obj-$(CONFIG_DRM_RADEON)+= radeon/
...

Changing amdkfd and radeon to late_initcall() however is
still the right call in orde to annotate explicitly a
delayed dependency requirement between the GPU drivers
and the IOMMUs.

We can't address the fragile nature of the link order
right now, but in the future that might be possible.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---

Please note, the changes to drivers/Makefile are just
for the sake of forcing the possible race to occur,
if this works well the actual [PATCH] submission will
skip those changes as its pointless to remove those
work arounds as it stands, due to the limited nature
of the levels available for addressing requirements.

Also, if you are aware of further dependency hell
things like these -- please do let me know as I am
interested in looking at addressing them.

 drivers/Makefile                        | 6 ++----
 drivers/gpu/drm/amd/amdkfd/kfd_module.c | 2 +-
 drivers/gpu/drm/radeon/radeon_drv.c     | 2 +-
 3 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/Makefile b/drivers/Makefile
index 0b6f3d60193d..0fbe3982041f 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -50,10 +50,7 @@ obj-$(CONFIG_RESET_CONTROLLER)	+= reset/
 obj-y				+= tty/
 obj-y				+= char/
 
-# iommu/ comes before gpu as gpu are using iommu controllers
-obj-$(CONFIG_IOMMU_SUPPORT)	+= iommu/
-
-# gpu/ comes after char for AGP vs DRM startup and after iommu
+# gpu/ comes after char for AGP vs DRM startup
 obj-y				+= gpu/
 
 obj-$(CONFIG_CONNECTOR)		+= connector/
@@ -147,6 +144,7 @@ obj-y				+= clk/
 
 obj-$(CONFIG_MAILBOX)		+= mailbox/
 obj-$(CONFIG_HWSPINLOCK)	+= hwspinlock/
+obj-$(CONFIG_IOMMU_SUPPORT)	+= iommu/
 obj-$(CONFIG_REMOTEPROC)	+= remoteproc/
 obj-$(CONFIG_RPMSG)		+= rpmsg/
 
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_module.c b/drivers/gpu/drm/amd/amdkfd/kfd_module.c
index 850a5623661f..3d1dab8a31c7 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_module.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_module.c
@@ -141,7 +141,7 @@ static void __exit kfd_module_exit(void)
 	dev_info(kfd_device, "Removed module\n");
 }
 
-module_init(kfd_module_init);
+late_initcall(kfd_module_init);
 module_exit(kfd_module_exit);
 
 MODULE_AUTHOR(KFD_DRIVER_AUTHOR);
diff --git a/drivers/gpu/drm/radeon/radeon_drv.c b/drivers/gpu/drm/radeon/radeon_drv.c
index b55aa740171f..1fa1b7f3a89c 100644
--- a/drivers/gpu/drm/radeon/radeon_drv.c
+++ b/drivers/gpu/drm/radeon/radeon_drv.c
@@ -609,7 +609,7 @@ static void __exit radeon_exit(void)
 	radeon_unregister_atpx_handler();
 }
 
-module_init(radeon_init);
+late_initcall(radeon_init);
 module_exit(radeon_exit);
 
 MODULE_AUTHOR(DRIVER_AUTHOR);
-- 
2.8.2

[toc] | [prev] | [next] | [standalone]

#1408583 — Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon

On Fri, May 27, 2016 at 4:18 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> To get KFD support in radeon we need the following
> initialization to happen in this order, their
> respective driver file that has its init routine
> listed next to it:
>
> 0. AMD IOMMUv1:    arch/x86/kernel/pci-dma.c
> 1. AMD IOMMUv2:    drivers/iommu/amd_iommu_v2.c
> 2. AMD KFD:        drivers/gpu/drm/amd/amdkfd/kfd_module.c
> 3. AMD Radeon:     drivers/gpu/drm/radeon/radeon_drv.c

Also AMD amdgpu (for VI+ APUs)
>
> Order is rather implicit, but these drivers can currently
> only do so much given the amount of leg room available.
> Below are the respective init routines and how they are
> initialized:
>
> arch/x86/kernel/pci-dma.c               rootfs_initcall(pci_iommu_init);
> drivers/iommu/amd_iommu_v2.c            module_init(amd_iommu_v2_init);
> drivers/gpu/drm/amd/amdkfd/kfd_module.c module_init(kfd_module_init);
> drivers/gpu/drm/radeon/radeon_drv.c     module_init(radeon_init);
>
> When a driver is built-in module_init() folds to use
> device_initcall(), and we have the following possible
> orders:
>
>         #define pure_initcall(fn)    __define_initcall(fn, 0)
>         #define core_initcall(fn)    __define_initcall(fn, 1)
>         #define postcore_initcall(fn)__define_initcall(fn, 2)
>         #define arch_initcall(fn)    __define_initcall(fn, 3)
>         #define subsys_initcall(fn)  __define_initcall(fn, 4)
>         #define fs_initcall(fn)      __define_initcall(fn, 5)
>         ---------------------------------------------------------
>         #define rootfs_initcall(fn)  __define_initcall(fn, rootfs)
>         #define device_initcall(fn)  __define_initcall(fn, 6)
>         #define late_initcall(fn)    __define_initcall(fn, 7)
>
> Since we start off from rootfs_initcall(), it gives us 3 more
> levels of leg room to play with for order semantics, this isn't
> enough to address all required levels of dependencies, this
> is specially true given that AMD-KFD needs to be loaded before
> the radeon driver -- -but this it not enforced by symbols.
> If the AMD-KFD driver is not loaded prior to the radeon driver
> because otherwise the radeon driver will not initialize the
> AMD-KFD driver and you get no KFD functionality in userspace.
>
> Commit 1bacc894c227fad8a7 ("drivers: Move iommu/ before gpu/ in
> Makefile") works around some of the possibe races between
> the AMD IOMMU v2 and GPU drivers by changing the link order.
> This is fragile, however its the bets we can do, given that
> making the GPU drivers use late_initcall() would also implicate
> a similar race between them. That possible race is fortunatley
> addressed given that the drm Makefile currently has amdkfd
> linked prior to radeon:
>
> drivers/gpu/drm/Makefile
> ...
> obj-$(CONFIG_HSA_AMD) += amd/amdkfd/
> obj-$(CONFIG_DRM_RADEON)+= radeon/
> ...
>
> Changing amdkfd and radeon to late_initcall() however is
> still the right call in orde to annotate explicitly a
> delayed dependency requirement between the GPU drivers
> and the IOMMUs.
>
> We can't address the fragile nature of the link order
> right now, but in the future that might be possible.
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>
> Please note, the changes to drivers/Makefile are just
> for the sake of forcing the possible race to occur,
> if this works well the actual [PATCH] submission will
> skip those changes as its pointless to remove those
> work arounds as it stands, due to the limited nature
> of the levels available for addressing requirements.
>
> Also, if you are aware of further dependency hell
> things like these -- please do let me know as I am
> interested in looking at addressing them.
>
>  drivers/Makefile                        | 6 ++----
>  drivers/gpu/drm/amd/amdkfd/kfd_module.c | 2 +-
>  drivers/gpu/drm/radeon/radeon_drv.c     | 2 +-
>  3 files changed, 4 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/Makefile b/drivers/Makefile
> index 0b6f3d60193d..0fbe3982041f 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -50,10 +50,7 @@ obj-$(CONFIG_RESET_CONTROLLER)       += reset/
>  obj-y                          += tty/
>  obj-y                          += char/
>
> -# iommu/ comes before gpu as gpu are using iommu controllers
> -obj-$(CONFIG_IOMMU_SUPPORT)    += iommu/
> -
> -# gpu/ comes after char for AGP vs DRM startup and after iommu
> +# gpu/ comes after char for AGP vs DRM startup
>  obj-y                          += gpu/
>
>  obj-$(CONFIG_CONNECTOR)                += connector/
> @@ -147,6 +144,7 @@ obj-y                               += clk/
>
>  obj-$(CONFIG_MAILBOX)          += mailbox/
>  obj-$(CONFIG_HWSPINLOCK)       += hwspinlock/
> +obj-$(CONFIG_IOMMU_SUPPORT)    += iommu/
>  obj-$(CONFIG_REMOTEPROC)       += remoteproc/
>  obj-$(CONFIG_RPMSG)            += rpmsg/
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_module.c b/drivers/gpu/drm/amd/amdkfd/kfd_module.c
> index 850a5623661f..3d1dab8a31c7 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_module.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_module.c
> @@ -141,7 +141,7 @@ static void __exit kfd_module_exit(void)
>         dev_info(kfd_device, "Removed module\n");
>  }
>
> -module_init(kfd_module_init);
> +late_initcall(kfd_module_init);
>  module_exit(kfd_module_exit);
>
>  MODULE_AUTHOR(KFD_DRIVER_AUTHOR);
> diff --git a/drivers/gpu/drm/radeon/radeon_drv.c b/drivers/gpu/drm/radeon/radeon_drv.c
> index b55aa740171f..1fa1b7f3a89c 100644
> --- a/drivers/gpu/drm/radeon/radeon_drv.c
> +++ b/drivers/gpu/drm/radeon/radeon_drv.c
> @@ -609,7 +609,7 @@ static void __exit radeon_exit(void)
>         radeon_unregister_atpx_handler();
>  }
>
> -module_init(radeon_init);
> +late_initcall(radeon_init);
>  module_exit(radeon_exit);

Need to modify also amdgpu module_init

>
>  MODULE_AUTHOR(DRIVER_AUTHOR);
> --
> 2.8.2
>

I tested this on Kaveri, and amdkfd is working. For amdkfd that's
fine, but IMO that's not enough testing for radeon/amdgpu. I would
like to hear AMD's developers take on this.

Oded

[toc] | [prev] | [next] | [standalone]

#1408630 — Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon

On Fri, May 27, 2016 at 3:18 AM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
> To get KFD support in radeon we need the following
> initialization to happen in this order, their
> respective driver file that has its init routine
> listed next to it:
>
> 0. AMD IOMMUv1:    arch/x86/kernel/pci-dma.c
> 1. AMD IOMMUv2:    drivers/iommu/amd_iommu_v2.c
> 2. AMD KFD:        drivers/gpu/drm/amd/amdkfd/kfd_module.c
> 3. AMD Radeon:     drivers/gpu/drm/radeon/radeon_drv.c
>
> Order is rather implicit, but these drivers can currently
> only do so much given the amount of leg room available.
> Below are the respective init routines and how they are
> initialized:
>
> arch/x86/kernel/pci-dma.c               rootfs_initcall(pci_iommu_init);
> drivers/iommu/amd_iommu_v2.c            module_init(amd_iommu_v2_init);
> drivers/gpu/drm/amd/amdkfd/kfd_module.c module_init(kfd_module_init);
> drivers/gpu/drm/radeon/radeon_drv.c     module_init(radeon_init);
>
> When a driver is built-in module_init() folds to use
> device_initcall(), and we have the following possible
> orders:
>
>         #define pure_initcall(fn)    __define_initcall(fn, 0)
>         #define core_initcall(fn)    __define_initcall(fn, 1)
>         #define postcore_initcall(fn)__define_initcall(fn, 2)
>         #define arch_initcall(fn)    __define_initcall(fn, 3)
>         #define subsys_initcall(fn)  __define_initcall(fn, 4)
>         #define fs_initcall(fn)      __define_initcall(fn, 5)
>         ---------------------------------------------------------
>         #define rootfs_initcall(fn)  __define_initcall(fn, rootfs)
>         #define device_initcall(fn)  __define_initcall(fn, 6)
>         #define late_initcall(fn)    __define_initcall(fn, 7)
>
> Since we start off from rootfs_initcall(), it gives us 3 more
> levels of leg room to play with for order semantics, this isn't
> enough to address all required levels of dependencies, this
> is specially true given that AMD-KFD needs to be loaded before
> the radeon driver -- -but this it not enforced by symbols.
> If the AMD-KFD driver is not loaded prior to the radeon driver
> because otherwise the radeon driver will not initialize the
> AMD-KFD driver and you get no KFD functionality in userspace.
>
> Commit 1bacc894c227fad8a7 ("drivers: Move iommu/ before gpu/ in
> Makefile") works around some of the possibe races between
> the AMD IOMMU v2 and GPU drivers by changing the link order.
> This is fragile, however its the bets we can do, given that
> making the GPU drivers use late_initcall() would also implicate
> a similar race between them. That possible race is fortunatley
> addressed given that the drm Makefile currently has amdkfd
> linked prior to radeon:
>
> drivers/gpu/drm/Makefile
> ...
> obj-$(CONFIG_HSA_AMD) += amd/amdkfd/
> obj-$(CONFIG_DRM_RADEON)+= radeon/
> ...
>
> Changing amdkfd and radeon to late_initcall() however is
> still the right call in orde to annotate explicitly a
> delayed dependency requirement between the GPU drivers
> and the IOMMUs.
>
> We can't address the fragile nature of the link order
> right now, but in the future that might be possible.
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>
> Please note, the changes to drivers/Makefile are just
> for the sake of forcing the possible race to occur,
> if this works well the actual [PATCH] submission will
> skip those changes as its pointless to remove those
> work arounds as it stands, due to the limited nature
> of the levels available for addressing requirements.
>
> Also, if you are aware of further dependency hell
> things like these -- please do let me know as I am
> interested in looking at addressing them.

This should be fixed with -EDEFER_PROBE instead. Frobbing initcall
levels should then just be done as an optimization to avoid too much
reprobing.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web