Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1265272 > unrolled thread

[PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-09 03:00 +0100
    Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Hannes Reinecke <hare@suse.de> - 2015-11-09 08:20 +0100
      Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-09 10:10 +0100
        Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-09 15:10 +0100
          Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-09 15:40 +0100
            Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Timur Tabi <timur@codeaurora.org> - 2015-11-10 00:30 +0100
            Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 00:30 +0100
              Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 09:50 +0100
                Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 17:10 +0100
                  Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 17:50 +0100
                    Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Timur Tabi <timur@codeaurora.org> - 2015-11-10 18:10 +0100
                      Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 20:20 +0100
                        Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Timur Tabi <timur@codeaurora.org> - 2015-11-10 22:10 +0100
                          Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Timur Tabi <timur@codeaurora.org> - 2015-11-10 23:00 +0100
                            Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 23:10 +0100
                          Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 23:00 +0100
                    Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 18:20 +0100
                      Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails James Bottomley <James.Bottomley@HansenPartnership.com> - 2015-11-10 19:30 +0100
                        Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 20:20 +0100
                          Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails James Bottomley <James.Bottomley@HansenPartnership.com> - 2015-11-10 20:50 +0100
                            Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 21:00 +0100
                              Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails James Bottomley <James.Bottomley@HansenPartnership.com> - 2015-11-10 21:10 +0100
                                Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 21:30 +0100
                                  Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails James Bottomley <James.Bottomley@HansenPartnership.com> - 2015-11-10 21:40 +0100
                      Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 21:00 +0100
                        Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-10 22:00 +0100
                          Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails Arnd Bergmann <arnd@arndb.de> - 2015-11-10 23:10 +0100
      Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA  fails Sinan Kaya <okaya@codeaurora.org> - 2015-11-09 15:10 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1266751 — Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

On 11/10/2015 2:43 PM, James Bottomley wrote:
> The Issue, as stated by LSI is
>
>          Initially set the consistent DMA mask to 32 bit and then change
>          it
>          to 64 bit mask after allocating RDPQ pools by calling the
>          function
>          _base_change_consistent_dma_mask. This is to ensure that all the
>          upper 32 bits of RDPQ entries's base address to be same.
>

Need somebody from mpt to confirm that this behavior is still valid for 
the recent cards besides altix.

> If you set a 64 bit coherent mask before this point, you're benefiting
> from being lucky that all the upper 32 bits of the allocations are the
> same ... we can't code a driver to rely on luck.  Particularly not when
> the failure mode looks like it would be silent and deadly.

Of course nobody wants unreliable code.

I'm wondering if I was just lucky during my testing or the 92xx and 93xx 
hardware supports full 64 bit range. I don't have any insight into what 
the endpoint does or what it is capable of.

>
>> >Another comment here from you.
>> >https://lkml.org/lkml/2015/4/2/28
>> >
>> >"Well, it was originally a hack for altix, because they had no regions
>> >below 4GB and had to specifically manufacture them.  As you know, in
>> >Linux, if Intel doesn't need it, no-one cares and the implementation
>> >bitrots."
>> >
>> >Maybe, it is time to fix the code for more recent (even decent) hardware?
> What do you mean "fix the code"?  The code isn't broken, it's
> parametrising issues with particular hardware.  There's no software work
> around (except allocating memory with the correct characteristics).

Need confirmation. I'm questioning if we are stuck with this behavior 
because of altix or something else. If the latter case, the code could 
have used PCI ID to do something special for it.

-- 
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266757 — Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

On Tue, 2015-11-10 at 14:56 -0500, Sinan Kaya wrote:
> 
> On 11/10/2015 2:43 PM, James Bottomley wrote:
> > The Issue, as stated by LSI is
> >
> >          Initially set the consistent DMA mask to 32 bit and then change
> >          it
> >          to 64 bit mask after allocating RDPQ pools by calling the
> >          function
> >          _base_change_consistent_dma_mask. This is to ensure that all the
> >          upper 32 bits of RDPQ entries's base address to be same.
> >
> 
> Need somebody from mpt to confirm that this behavior is still valid for 
> the recent cards besides altix.

OK, you don't seem to be understanding the problem: the Altix isn't a
LSI card, it was a SGI platform.  It was the platform where we first
discovered the issue that a lot of storage cards didn't work because it
by default had no memory below 4GB.  The reason coherent masks were
introduced was initially so the Altix could manufacture and manage a
region of memory in the lower 4GB region and we would guarantee to make
allocations from it so the storage cards would then work on that
platform.

I thought the Altix was a historical relic because after they
disappeared, there was no other platform with this issue ...  until you
came along.

James


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266762 — Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

On 11/10/2015 3:05 PM, James Bottomley wrote:
> OK, you don't seem to be understanding the problem: the Altix isn't a
> LSI card, it was a SGI platform.

Got it.

> It was the platform where we first
> discovered the issue that a lot of storage cards didn't work because it
> by default had no memory below 4GB.  The reason coherent masks were
> introduced was initially so the Altix could manufacture and manage a
> region of memory in the lower 4GB region and we would guarantee to make
> allocations from it so the storage cards would then work on that
> platform.

I can't fix the issue if the card cannot do 64 bit DMA when IOMMU is not 
there. I need IOMMU enabled all the time for this card.

-- 
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266767 — Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

On Tue, 2015-11-10 at 15:26 -0500, Sinan Kaya wrote:
> 
> On 11/10/2015 3:05 PM, James Bottomley wrote:
> > OK, you don't seem to be understanding the problem: the Altix isn't a
> > LSI card, it was a SGI platform.
> 
> Got it.
> 
> > It was the platform where we first
> > discovered the issue that a lot of storage cards didn't work because it
> > by default had no memory below 4GB.  The reason coherent masks were
> > introduced was initially so the Altix could manufacture and manage a
> > region of memory in the lower 4GB region and we would guarantee to make
> > allocations from it so the storage cards would then work on that
> > platform.
> 
> I can't fix the issue if the card cannot do 64 bit DMA when IOMMU is not 
> there. I need IOMMU enabled all the time for this card.

That depends on the limitations of your platform.  The Altix only used
an iommu to manufacture the coherent memory for the descriptors, but the
card itself mostly operated in bypass mode (using 64 bit physical
addresses rather than iommu remapped ones), so all accesses except for
the few firmware descriptor ones didn't use an iommu.  Apparently this
was for performance reasons.

So, to recap, the card itself *can* do 64 bit DMA.  The limitation is
that the RDPQ descriptors need to be all in the same region of 4G memory
and the way the driver ensures this is to set the 64 bit coherent mask
after using the 32 bit one to allocate the RDPQ pools.

James


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266754

On Tuesday 10 November 2015 12:19:33 Sinan Kaya wrote:
> On 11/10/2015 11:47 AM, Arnd Bergmann wrote:
> > On Tuesday 10 November 2015 11:06:40 Sinan Kaya wrote:
> >> On 11/10/2015 3:38 AM, Arnd Bergmann wrote:
> >>   > No, as Timur found, the driver is correct and it intentionally
> >>> sets the 32-bit mask, and that is guaranteed to work on all sane
> >>> hardware. Don't change the driver but find a better platform for
> >>> your workload, or talk to the people that are responsible for
> >>> the platform and get them to fix it.
> >>
> >> Platform does have an IOMMU. No issues there. I am trying to clean out
> >> the patch pipe I have in order to get this card working with and without
> >> IOMMU.
> >
> > On PowerPC, I think we automatically enable the IOMMU whenever a DMA
> > mask is set that doesn't cover all of the RAM. We could think about
> > doing the same thing on ARM64 to make all devices work out of the box.
> >
> 
> The ACPI IORT table declares whether you enable IOMMU for a particular 
> device or not. The placement of IOMMU HW is system specific. The IORT 
> table gives the IOMMU HW topology to the operating system.

This sounds odd. Clearly you need to specify the IOMMU settings for each
possible PCI device independent of whether the OS actually uses the IOMMU
or not. In a lot of cases, we want to turn it off to get better performance
when the driver has set a DMA mask that covers all of RAM, but you
also want to enable the IOMMU for debugging purposes or for device
assignment if you run virtual machines. The bootloader doesn't know how
the device is going to be used, so it cannot define the policy here.

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266784 — Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

On 11/10/2015 2:56 PM, Arnd Bergmann wrote:
>> The ACPI IORT table declares whether you enable IOMMU for a particular
>> >device or not. The placement of IOMMU HW is system specific. The IORT
>> >table gives the IOMMU HW topology to the operating system.
> This sounds odd. Clearly you need to specify the IOMMU settings for each
> possible PCI device independent of whether the OS actually uses the IOMMU
> or not.

There are provisions to have DMA mask in the PCIe host bridge not at the 
PCIe device level inside IORT table. This setting is specific for each 
PCIe bus. It is not per PCIe device.

It is assumed that the endpoint device driver knows the hardware for 
PCIe devices. The driver can also query the supported DMA bits by this 
platform via DMA APIs and will request the correct DMA mask from the DMA 
subsystem (surprise!).

>In a lot of cases, we want to turn it off to get better performance
> when the driver has set a DMA mask that covers all of RAM, but you
> also want to enable the IOMMU for debugging purposes or for device
> assignment if you run virtual machines. The bootloader doesn't know how
> the device is going to be used, so it cannot define the policy here.

I think we'll end up adding a virtualization option to the UEFI BIOS 
similar to how Intel platforms work. Based on this switch, we'll end up 
patching the ACPI table.

If I remove the IORT entry, then the device is in coherent mode with 
device accessing the full RAM range.

If I have the IORT table, the device is in IOMMU translation mode.

Details are in the IORT spec.


-- 
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1266810

On Tuesday 10 November 2015 15:58:19 Sinan Kaya wrote:
> 
> On 11/10/2015 2:56 PM, Arnd Bergmann wrote:
> >> The ACPI IORT table declares whether you enable IOMMU for a particular
> >> >device or not. The placement of IOMMU HW is system specific. The IORT
> >> >table gives the IOMMU HW topology to the operating system.
> > This sounds odd. Clearly you need to specify the IOMMU settings for each
> > possible PCI device independent of whether the OS actually uses the IOMMU
> > or not.
> 
> There are provisions to have DMA mask in the PCIe host bridge not at the 
> PCIe device level inside IORT table. This setting is specific for each 
> PCIe bus. It is not per PCIe device.

Same thing, I meant the bootloader must provide all the information that
is needed to use the IOMMU on all PCI devices. I don't care where the IOMMU
driver gets that information. Some IOMMUs require programming a
bus/device/function specific number into the I/O page tables, and they
might not always have the same algorithm to map from the PCI numbers
into their own number space.

> It is assumed that the endpoint device driver knows the hardware for 
> PCIe devices. The driver can also query the supported DMA bits by this 
> platform via DMA APIs and will request the correct DMA mask from the DMA 
> subsystem (surprise!).

I know how the negotiation works. Note that dma_get_required_mask()
will only tell you what mask the device needs to access all of memory,
while both the device and bus may have additional limitations, and
there is not always a solution.

> >In a lot of cases, we want to turn it off to get better performance
> > when the driver has set a DMA mask that covers all of RAM, but you
> > also want to enable the IOMMU for debugging purposes or for device
> > assignment if you run virtual machines. The bootloader doesn't know how
> > the device is going to be used, so it cannot define the policy here.
> 
> I think we'll end up adding a virtualization option to the UEFI BIOS 
> similar to how Intel platforms work. Based on this switch, we'll end up 
> patching the ACPI table.
> 
> If I remove the IORT entry, then the device is in coherent mode with 
> device accessing the full RAM range.
> 
> If I have the IORT table, the device is in IOMMU translation mode.
> 
> Details are in the IORT spec.

I think that would suck a lot more than being slightly out of spec
regarding SBSA if you make the low PCI addresses map to the start
of RAM. Asking users to select a 'virtualization' option based on
what kind of PCI device and kernel version they have is a major
hassle.

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1265691 — Re: [PATCH V2 1/3] scsi: mptxsas: try 64 bit DMA when 32 bit DMA fails

On 11/9/2015 2:09 AM, Hannes Reinecke wrote:
> On 11/09/2015 02:57 AM, Sinan Kaya wrote:
>> Current code gives up when 32 bit DMA is not supported.
>> This problem has been observed on systems without any
>> memory below 4 gig.
>>
>> This patch tests 64 bit support before bailing out to find
>> a working combination.
>>
> That feels decidedly odd.
>
> Why do you probe for 64bit if 32bit fails?
> Typically it's the other way round, on the grounds that 64bit DMA
> should be preferred over 32bit.
> Can you explain why it needs to be done the other way round here?
>
> Cheers,
>
> Hannes
>

The platform does not have any memory below 4G. So, 32 bit DMA is not 
possible. I'm trying to use 64 bit DMA instead since both the platform 
and hardware supports it. Current code will not try 64 bit DMA if 32 bit 
DMA is not working.

-- 
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web