Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1401413 > unrolled thread

ioatdma(Intel(R) I/OAT DMA Engine init failed)

Started byGavin Guo <gavin.guo@canonical.com>
First post2016-05-16 12:10 +0200
Last post2016-05-25 09:00 +0200
Articles 9 — 4 participants

Back to article view | Back to linux.kernel

Contents

  ioatdma(Intel(R) I/OAT DMA Engine init failed) Gavin Guo <gavin.guo@canonical.com> - 2016-05-16 12:10 +0200
    Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) Vinod Koul <vinod.koul@intel.com> - 2016-05-17 12:10 +0200
      Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) Gavin Guo <gavin.guo@canonical.com> - 2016-05-18 15:30 +0200
        Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) "Jiang, Dave" <dave.jiang@intel.com> - 2016-05-18 18:50 +0200
          RE: ioatdma(Intel(R) I/OAT DMA Engine init failed) "Jiang, Dave" <dave.jiang@intel.com> - 2016-05-19 16:50 +0200
            Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) Vinod Koul <vinod.koul@intel.com> - 2016-05-19 19:20 +0200
              RE: ioatdma(Intel(R) I/OAT DMA Engine init failed) "Jiang, Dave" <dave.jiang@intel.com> - 2016-05-19 22:20 +0200
                Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) Yinghai Lu <yinghai@kernel.org> - 2016-05-20 00:20 +0200
                  Re: ioatdma(Intel(R) I/OAT DMA Engine init failed) Gavin Guo <gavin.guo@canonical.com> - 2016-05-25 09:00 +0200

#1401413 — ioatdma(Intel(R) I/OAT DMA Engine init failed)

FromGavin Guo <gavin.guo@canonical.com>
Date2016-05-16 12:10 +0200
Subjectioatdma(Intel(R) I/OAT DMA Engine init failed)
Message-ID<rznDj-5Bk-11@gated-at.bofh.it>
The following error messages can be observed on the Intel Haswell-E
chipset with v3.13 kernel. After the analysis, I found there is no
difference in the logic of these error messages in the current
upstream kernel. I also searched the git log and can't find any commit
which is fix to the error(correct me if I am wrong). The following is
the detail, and I'll really appreciate if there is any comment. :)

ioatdma 0000:00:04.0: channel error register unreachable
ioatdma 0000:00:04.0: channel enumeration error
ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
ioatdma 0000:00:04.1: channel error register unreachable
ioatdma 0000:00:04.1: channel enumeration error
ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
...
ioatdma 0000:00:04.7: channel error register unreachable
ioatdma 0000:00:04.7: channel enumeration error
ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
mei_me 0000:00:16.0: initialization failed.

There are 8 I/OAT DMA controllers on the Haswell-E chipset:
8086:2f20 ~ 8086:2f27
80:04.0 System peripheral: Intel Corporation Haswell-E DMA Channel 0 (rev 02)
80:04.1 System peripheral: Intel Corporation Haswell-E DMA Channel 1 (rev 02)
80:04.2 System peripheral: Intel Corporation Haswell-E DMA Channel 2 (rev 02)
80:04.3 System peripheral: Intel Corporation Haswell-E DMA Channel 3 (rev 02)
80:04.4 System peripheral: Intel Corporation Haswell-E DMA Channel 4 (rev 02)
80:04.5 System peripheral: Intel Corporation Haswell-E DMA Channel 5 (rev 02)
80:04.6 System peripheral: Intel Corporation Haswell-E DMA Channel 6 (rev 02)
80:04.7 System peripheral: Intel Corporation Haswell-E DMA Channel 7 (rev 02)

Analysis:
The bug happens when the driver is resetting DMA controller, this is
the sequence: The function, ioat_pci_probe, is called when the DMA
controller is detected by the PCI bus. Then,
ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
ioat3_reset_hw. The following code can be found in the ioat3_reset_hw:

drivers/dma/ioat/dma_v3.c:
        chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
        writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
...
        err = pci_read_config_dword(pdev,
IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
if (err) {
dev_err(&pdev->dev,
"channel error register unreachable\n");
return err;
}

Obviously, there are something wrong in the channel error register
reset process. Then all the way back to ioat_probe(). Because the
error happens, the dma->chancnt will be set to 0:

drivers/dma/ioat/dma.c:
        if (!dma->chancnt) {
                dev_err(dev, "channel enumeration error\n");
                goto err_setup_interrupts;
        }

Finally back to ioat_pci_probe:

drivers/dma/ioat/pci.c:
                err = ioat3_dma_probe(device, ioat_dca_enabled);
        else
                return -ENODEV;

        if (err) {
                dev_err(dev, "Intel(R) I/OAT DMA Engine init
failed\n");
                return -ENODEV;

[toc] | [next] | [standalone]

#1402259

FromVinod Koul <vinod.koul@intel.com>
Date2016-05-17 12:10 +0200
Message-ID<rzK6S-3cU-31@gated-at.bofh.it>
In reply to#1401413
On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote:
> The following error messages can be observed on the Intel Haswell-E
> chipset with v3.13 kernel. After the analysis, I found there is no
> difference in the logic of these error messages in the current
> upstream kernel. I also searched the git log and can't find any commit
> which is fix to the error(correct me if I am wrong). The following is
> the detail, and I'll really appreciate if there is any comment. :)

3.13 is ancient, can you check this on latest kernel

> 
> ioatdma 0000:00:04.0: channel error register unreachable
> ioatdma 0000:00:04.0: channel enumeration error
> ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
> ioatdma 0000:00:04.1: channel error register unreachable
> ioatdma 0000:00:04.1: channel enumeration error
> ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
> ...
> ioatdma 0000:00:04.7: channel error register unreachable
> ioatdma 0000:00:04.7: channel enumeration error
> ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
> mei_me 0000:00:16.0: initialization failed.
> 
> There are 8 I/OAT DMA controllers on the Haswell-E chipset:
> 8086:2f20 ~ 8086:2f27
> 80:04.0 System peripheral: Intel Corporation Haswell-E DMA Channel 0 (rev 02)
> 80:04.1 System peripheral: Intel Corporation Haswell-E DMA Channel 1 (rev 02)
> 80:04.2 System peripheral: Intel Corporation Haswell-E DMA Channel 2 (rev 02)
> 80:04.3 System peripheral: Intel Corporation Haswell-E DMA Channel 3 (rev 02)
> 80:04.4 System peripheral: Intel Corporation Haswell-E DMA Channel 4 (rev 02)
> 80:04.5 System peripheral: Intel Corporation Haswell-E DMA Channel 5 (rev 02)
> 80:04.6 System peripheral: Intel Corporation Haswell-E DMA Channel 6 (rev 02)
> 80:04.7 System peripheral: Intel Corporation Haswell-E DMA Channel 7 (rev 02)
> 
> Analysis:
> The bug happens when the driver is resetting DMA controller, this is
> the sequence: The function, ioat_pci_probe, is called when the DMA
> controller is detected by the PCI bus. Then,
> ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
> ioat3_reset_hw. The following code can be found in the ioat3_reset_hw:
> 
> drivers/dma/ioat/dma_v3.c:
>         chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
>         writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
> ...
>         err = pci_read_config_dword(pdev,
> IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
> if (err) {
> dev_err(&pdev->dev,
> "channel error register unreachable\n");
> return err;
> }
> 
> Obviously, there are something wrong in the channel error register
> reset process. Then all the way back to ioat_probe(). Because the
> error happens, the dma->chancnt will be set to 0:
> 
> drivers/dma/ioat/dma.c:
>         if (!dma->chancnt) {
>                 dev_err(dev, "channel enumeration error\n");
>                 goto err_setup_interrupts;
>         }
> 
> Finally back to ioat_pci_probe:
> 
> drivers/dma/ioat/pci.c:
>                 err = ioat3_dma_probe(device, ioat_dca_enabled);
>         else
>                 return -ENODEV;
> 
>         if (err) {
>                 dev_err(dev, "Intel(R) I/OAT DMA Engine init
> failed\n");
>                 return -ENODEV;

-- 
~Vinod

[toc] | [prev] | [next] | [standalone]

#1402975

FromGavin Guo <gavin.guo@canonical.com>
Date2016-05-18 15:30 +0200
Message-ID<rA9HX-2GR-15@gated-at.bofh.it>
In reply to#1402259
On Tue, May 17, 2016 at 6:06 PM, Vinod Koul <vinod.koul@intel.com> wrote:
> On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote:
>> The following error messages can be observed on the Intel Haswell-E
>> chipset with v3.13 kernel. After the analysis, I found there is no
>> difference in the logic of these error messages in the current
>> upstream kernel. I also searched the git log and can't find any commit
>> which is fix to the error(correct me if I am wrong). The following is
>> the detail, and I'll really appreciate if there is any comment. :)
>
> 3.13 is ancient, can you check this on latest kernel

Thank you for the comment. It's running on the production system. However,
I'll try to figure out if it's possible to test the latest kernel.

>
>>
>> ioatdma 0000:00:04.0: channel error register unreachable
>> ioatdma 0000:00:04.0: channel enumeration error
>> ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
>> ioatdma 0000:00:04.1: channel error register unreachable
>> ioatdma 0000:00:04.1: channel enumeration error
>> ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
>> ...
>> ioatdma 0000:00:04.7: channel error register unreachable
>> ioatdma 0000:00:04.7: channel enumeration error
>> ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
>> mei_me 0000:00:16.0: initialization failed.
>>
>> There are 8 I/OAT DMA controllers on the Haswell-E chipset:
>> 8086:2f20 ~ 8086:2f27
>> 80:04.0 System peripheral: Intel Corporation Haswell-E DMA Channel 0 (rev 02)
>> 80:04.1 System peripheral: Intel Corporation Haswell-E DMA Channel 1 (rev 02)
>> 80:04.2 System peripheral: Intel Corporation Haswell-E DMA Channel 2 (rev 02)
>> 80:04.3 System peripheral: Intel Corporation Haswell-E DMA Channel 3 (rev 02)
>> 80:04.4 System peripheral: Intel Corporation Haswell-E DMA Channel 4 (rev 02)
>> 80:04.5 System peripheral: Intel Corporation Haswell-E DMA Channel 5 (rev 02)
>> 80:04.6 System peripheral: Intel Corporation Haswell-E DMA Channel 6 (rev 02)
>> 80:04.7 System peripheral: Intel Corporation Haswell-E DMA Channel 7 (rev 02)
>>
>> Analysis:
>> The bug happens when the driver is resetting DMA controller, this is
>> the sequence: The function, ioat_pci_probe, is called when the DMA
>> controller is detected by the PCI bus. Then,
>> ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
>> ioat3_reset_hw. The following code can be found in the ioat3_reset_hw:
>>
>> drivers/dma/ioat/dma_v3.c:
>>         chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
>>         writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
>> ...
>>         err = pci_read_config_dword(pdev,
>> IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
>> if (err) {
>> dev_err(&pdev->dev,
>> "channel error register unreachable\n");
>> return err;
>> }
>>
>> Obviously, there are something wrong in the channel error register
>> reset process. Then all the way back to ioat_probe(). Because the
>> error happens, the dma->chancnt will be set to 0:
>>
>> drivers/dma/ioat/dma.c:
>>         if (!dma->chancnt) {
>>                 dev_err(dev, "channel enumeration error\n");
>>                 goto err_setup_interrupts;
>>         }
>>
>> Finally back to ioat_pci_probe:
>>
>> drivers/dma/ioat/pci.c:
>>                 err = ioat3_dma_probe(device, ioat_dca_enabled);
>>         else
>>                 return -ENODEV;
>>
>>         if (err) {
>>                 dev_err(dev, "Intel(R) I/OAT DMA Engine init
>> failed\n");
>>                 return -ENODEV;
>
> --
> ~Vinod

[toc] | [prev] | [next] | [standalone]

#1403148

From"Jiang, Dave" <dave.jiang@intel.com>
Date2016-05-18 18:50 +0200
Message-ID<rAcPx-4AL-37@gated-at.bofh.it>
In reply to#1402975
On Wed, 2016-05-18 at 13:27 +0000, Gavin Guo wrote:
> On Tue, May 17, 2016 at 6:06 PM, Vinod Koul <vinod.koul@intel.com>
> wrote:
> > 
> > On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote:
> > > 
> > > The following error messages can be observed on the Intel
> > > Haswell-E
> > > chipset with v3.13 kernel. After the analysis, I found there is
> > > no
> > > difference in the logic of these error messages in the current
> > > upstream kernel. I also searched the git log and can't find any
> > > commit
> > > which is fix to the error(correct me if I am wrong). The
> > > following is
> > > the detail, and I'll really appreciate if there is any comment.
> > > :)
> > 3.13 is ancient, can you check this on latest kernel
> Thank you for the comment. It's running on the production system.
> However,
> I'll try to figure out if it's possible to test the latest kernel.

I wonder if you don't have the extended PCI config space access enabled
in your kernel config. 

> 
> > 
> > 
> > > 
> > > 
> > > ioatdma 0000:00:04.0: channel error register unreachable
> > > ioatdma 0000:00:04.0: channel enumeration error
> > > ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
> > > ioatdma 0000:00:04.1: channel error register unreachable
> > > ioatdma 0000:00:04.1: channel enumeration error
> > > ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
> > > ...
> > > ioatdma 0000:00:04.7: channel error register unreachable
> > > ioatdma 0000:00:04.7: channel enumeration error
> > > ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
> > > mei_me 0000:00:16.0: initialization failed.
> > > 
> > > There are 8 I/OAT DMA controllers on the Haswell-E chipset:
> > > 8086:2f20 ~ 8086:2f27
> > > 80:04.0 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 0 (rev 02)
> > > 80:04.1 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 1 (rev 02)
> > > 80:04.2 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 2 (rev 02)
> > > 80:04.3 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 3 (rev 02)
> > > 80:04.4 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 4 (rev 02)
> > > 80:04.5 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 5 (rev 02)
> > > 80:04.6 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 6 (rev 02)
> > > 80:04.7 System peripheral: Intel Corporation Haswell-E DMA
> > > Channel 7 (rev 02)
> > > 
> > > Analysis:
> > > The bug happens when the driver is resetting DMA controller, this
> > > is
> > > the sequence: The function, ioat_pci_probe, is called when the
> > > DMA
> > > controller is detected by the PCI bus. Then,
> > > ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
> > > ioat3_reset_hw. The following code can be found in the
> > > ioat3_reset_hw:
> > > 
> > > drivers/dma/ioat/dma_v3.c:
> > >         chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
> > >         writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
> > > ...
> > >         err = pci_read_config_dword(pdev,
> > > IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
> > > if (err) {
> > > dev_err(&pdev->dev,
> > > "channel error register unreachable\n");
> > > return err;
> > > }
> > > 
> > > Obviously, there are something wrong in the channel error
> > > register
> > > reset process. Then all the way back to ioat_probe(). Because the
> > > error happens, the dma->chancnt will be set to 0:
> > > 
> > > drivers/dma/ioat/dma.c:
> > >         if (!dma->chancnt) {
> > >                 dev_err(dev, "channel enumeration error\n");
> > >                 goto err_setup_interrupts;
> > >         }
> > > 
> > > Finally back to ioat_pci_probe:
> > > 
> > > drivers/dma/ioat/pci.c:
> > >                 err = ioat3_dma_probe(device, ioat_dca_enabled);
> > >         else
> > >                 return -ENODEV;
> > > 
> > >         if (err) {
> > >                 dev_err(dev, "Intel(R) I/OAT DMA Engine init
> > > failed\n");
> > >                 return -ENODEV;
> > --
> > ~Vinod

[toc] | [prev] | [next] | [standalone]

#1403791

From"Jiang, Dave" <dave.jiang@intel.com>
Date2016-05-19 16:50 +0200
Message-ID<rAxqV-12J-7@gated-at.bofh.it>
In reply to#1403148
> -----Original Message-----
> From: Gavin Guo [mailto:gavin.guo@canonical.com]
> Sent: Wednesday, May 18, 2016 8:19 PM
> To: Jiang, Dave <dave.jiang@intel.com>
> Cc: Koul, Vinod <vinod.koul@intel.com>; dmaengine@vger.kernel.org; linux-kernel@vger.kernel.org; Williams, Dan J
> <dan.j.williams@intel.com>
> Subject: Re: ioatdma(Intel(R) I/OAT DMA Engine init failed)
> 
> On Thu, May 19, 2016 at 12:49 AM, Jiang, Dave <dave.jiang@intel.com> wrote:
> > On Wed, 2016-05-18 at 13:27 +0000, Gavin Guo wrote:
> >> On Tue, May 17, 2016 at 6:06 PM, Vinod Koul <vinod.koul@intel.com>
> >> wrote:
> >> >
> >> > On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote:
> >> > >
> >> > > The following error messages can be observed on the Intel
> >> > > Haswell-E
> >> > > chipset with v3.13 kernel. After the analysis, I found there is
> >> > > no
> >> > > difference in the logic of these error messages in the current
> >> > > upstream kernel. I also searched the git log and can't find any
> >> > > commit
> >> > > which is fix to the error(correct me if I am wrong). The
> >> > > following is
> >> > > the detail, and I'll really appreciate if there is any comment.
> >> > > :)
> >> > 3.13 is ancient, can you check this on latest kernel
> >> Thank you for the comment. It's running on the production system.
> >> However,
> >> I'll try to figure out if it's possible to test the latest kernel.
> >
> > I wonder if you don't have the extended PCI config space access enabled
> > in your kernel config.
> 
> Really thanks for your advice. :)
> 
> I searched the internet about the extended PCI config space and found
> the link:
> 
> [Patch v2] Make PCI extended config space (MMCONFIG) a driver opt-in
> http://lwn.net/Articles/263288/

Can you try calling pci_enable_ext_config() in the PCI probe for your kernel? I just haven't seen this issue in the latest kernel. 

> 
> And I checked the config and found the CONFIG_PCI_MMCONFIG=y. The
> following string also can be observed in the dmesg:
> 
> [    1.419853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at
> [mem0x80000000-0x8fffffff] (base 0x80000000)
> [    1.419855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
> 
> It seems the extended PCI config space is enabled. If there is
> anything missed?
> 
> >
> >>
> >> >
> >> >
> >> > >
> >> > >
> >> > > ioatdma 0000:00:04.0: channel error register unreachable
> >> > > ioatdma 0000:00:04.0: channel enumeration error
> >> > > ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
> >> > > ioatdma 0000:00:04.1: channel error register unreachable
> >> > > ioatdma 0000:00:04.1: channel enumeration error
> >> > > ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
> >> > > ...
> >> > > ioatdma 0000:00:04.7: channel error register unreachable
> >> > > ioatdma 0000:00:04.7: channel enumeration error
> >> > > ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
> >> > > mei_me 0000:00:16.0: initialization failed.
> >> > >
> >> > > There are 8 I/OAT DMA controllers on the Haswell-E chipset:
> >> > > 8086:2f20 ~ 8086:2f27
> >> > > 80:04.0 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 0 (rev 02)
> >> > > 80:04.1 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 1 (rev 02)
> >> > > 80:04.2 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 2 (rev 02)
> >> > > 80:04.3 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 3 (rev 02)
> >> > > 80:04.4 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 4 (rev 02)
> >> > > 80:04.5 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 5 (rev 02)
> >> > > 80:04.6 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 6 (rev 02)
> >> > > 80:04.7 System peripheral: Intel Corporation Haswell-E DMA
> >> > > Channel 7 (rev 02)
> >> > >
> >> > > Analysis:
> >> > > The bug happens when the driver is resetting DMA controller, this
> >> > > is
> >> > > the sequence: The function, ioat_pci_probe, is called when the
> >> > > DMA
> >> > > controller is detected by the PCI bus. Then,
> >> > > ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
> >> > > ioat3_reset_hw. The following code can be found in the
> >> > > ioat3_reset_hw:
> >> > >
> >> > > drivers/dma/ioat/dma_v3.c:
> >> > >         chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
> >> > >         writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
> >> > > ...
> >> > >         err = pci_read_config_dword(pdev,
> >> > > IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
> >> > > if (err) {
> >> > > dev_err(&pdev->dev,
> >> > > "channel error register unreachable\n");
> >> > > return err;
> >> > > }
> >> > >
> >> > > Obviously, there are something wrong in the channel error
> >> > > register
> >> > > reset process. Then all the way back to ioat_probe(). Because the
> >> > > error happens, the dma->chancnt will be set to 0:
> >> > >
> >> > > drivers/dma/ioat/dma.c:
> >> > >         if (!dma->chancnt) {
> >> > >                 dev_err(dev, "channel enumeration error\n");
> >> > >                 goto err_setup_interrupts;
> >> > >         }
> >> > >
> >> > > Finally back to ioat_pci_probe:
> >> > >
> >> > > drivers/dma/ioat/pci.c:
> >> > >                 err = ioat3_dma_probe(device, ioat_dca_enabled);
> >> > >         else
> >> > >                 return -ENODEV;
> >> > >
> >> > >         if (err) {
> >> > >                 dev_err(dev, "Intel(R) I/OAT DMA Engine init
> >> > > failed\n");
> >> > >                 return -ENODEV;
> >> > --
> >> > ~Vinod

[toc] | [prev] | [next] | [standalone]

#1403874

FromVinod Koul <vinod.koul@intel.com>
Date2016-05-19 19:20 +0200
Message-ID<rAzM6-2EV-21@gated-at.bofh.it>
In reply to#1403791
On Thu, May 19, 2016 at 08:19:30PM +0530, Jiang, Dave wrote:
> > -----Original Message-----
> > From: Gavin Guo [mailto:gavin.guo@canonical.com]
> > Sent: Wednesday, May 18, 2016 8:19 PM
> > To: Jiang, Dave <dave.jiang@intel.com>
> > Cc: Koul, Vinod <vinod.koul@intel.com>; dmaengine@vger.kernel.org; linux-kernel@vger.kernel.org; Williams, Dan J
> > <dan.j.williams@intel.com>
> > Subject: Re: ioatdma(Intel(R) I/OAT DMA Engine init failed)
> > 
> > On Thu, May 19, 2016 at 12:49 AM, Jiang, Dave <dave.jiang@intel.com> wrote:
> > > On Wed, 2016-05-18 at 13:27 +0000, Gavin Guo wrote:
> > >> On Tue, May 17, 2016 at 6:06 PM, Vinod Koul <vinod.koul@intel.com>
> > >> wrote:
> > >> >
> > >> > On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote:
> > >> > >
> > >> > > The following error messages can be observed on the Intel
> > >> > > Haswell-E
> > >> > > chipset with v3.13 kernel. After the analysis, I found there is
> > >> > > no
> > >> > > difference in the logic of these error messages in the current
> > >> > > upstream kernel. I also searched the git log and can't find any
> > >> > > commit
> > >> > > which is fix to the error(correct me if I am wrong). The
> > >> > > following is
> > >> > > the detail, and I'll really appreciate if there is any comment.
> > >> > > :)
> > >> > 3.13 is ancient, can you check this on latest kernel
> > >> Thank you for the comment. It's running on the production system.
> > >> However,
> > >> I'll try to figure out if it's possible to test the latest kernel.
> > >
> > > I wonder if you don't have the extended PCI config space access enabled
> > > in your kernel config.
> > 
> > Really thanks for your advice. :)
> > 
> > I searched the internet about the extended PCI config space and found
> > the link:
> > 
> > [Patch v2] Make PCI extended config space (MMCONFIG) a driver opt-in
> > http://lwn.net/Articles/263288/
> 
> Can you try calling pci_enable_ext_config() in the PCI probe for your kernel? I just haven't seen this issue in the latest kernel. 

Do we need that to be called explicitly by driver, should that not be enabled
by default?

> 
> > 
> > And I checked the config and found the CONFIG_PCI_MMCONFIG=y. The
> > following string also can be observed in the dmesg:
> > 
> > [    1.419853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at
> > [mem0x80000000-0x8fffffff] (base 0x80000000)
> > [    1.419855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
> > 
> > It seems the extended PCI config space is enabled. If there is
> > anything missed?
> > 
> > >
> > >>
> > >> >
> > >> >
> > >> > >
> > >> > >
> > >> > > ioatdma 0000:00:04.0: channel error register unreachable
> > >> > > ioatdma 0000:00:04.0: channel enumeration error
> > >> > > ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
> > >> > > ioatdma 0000:00:04.1: channel error register unreachable
> > >> > > ioatdma 0000:00:04.1: channel enumeration error
> > >> > > ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
> > >> > > ...
> > >> > > ioatdma 0000:00:04.7: channel error register unreachable
> > >> > > ioatdma 0000:00:04.7: channel enumeration error
> > >> > > ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
> > >> > > mei_me 0000:00:16.0: initialization failed.
> > >> > >
> > >> > > There are 8 I/OAT DMA controllers on the Haswell-E chipset:
> > >> > > 8086:2f20 ~ 8086:2f27
> > >> > > 80:04.0 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 0 (rev 02)
> > >> > > 80:04.1 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 1 (rev 02)
> > >> > > 80:04.2 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 2 (rev 02)
> > >> > > 80:04.3 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 3 (rev 02)
> > >> > > 80:04.4 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 4 (rev 02)
> > >> > > 80:04.5 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 5 (rev 02)
> > >> > > 80:04.6 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 6 (rev 02)
> > >> > > 80:04.7 System peripheral: Intel Corporation Haswell-E DMA
> > >> > > Channel 7 (rev 02)
> > >> > >
> > >> > > Analysis:
> > >> > > The bug happens when the driver is resetting DMA controller, this
> > >> > > is
> > >> > > the sequence: The function, ioat_pci_probe, is called when the
> > >> > > DMA
> > >> > > controller is detected by the PCI bus. Then,
> > >> > > ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
> > >> > > ioat3_reset_hw. The following code can be found in the
> > >> > > ioat3_reset_hw:
> > >> > >
> > >> > > drivers/dma/ioat/dma_v3.c:
> > >> > >         chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
> > >> > >         writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
> > >> > > ...
> > >> > >         err = pci_read_config_dword(pdev,
> > >> > > IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
> > >> > > if (err) {
> > >> > > dev_err(&pdev->dev,
> > >> > > "channel error register unreachable\n");
> > >> > > return err;
> > >> > > }
> > >> > >
> > >> > > Obviously, there are something wrong in the channel error
> > >> > > register
> > >> > > reset process. Then all the way back to ioat_probe(). Because the
> > >> > > error happens, the dma->chancnt will be set to 0:
> > >> > >
> > >> > > drivers/dma/ioat/dma.c:
> > >> > >         if (!dma->chancnt) {
> > >> > >                 dev_err(dev, "channel enumeration error\n");
> > >> > >                 goto err_setup_interrupts;
> > >> > >         }
> > >> > >
> > >> > > Finally back to ioat_pci_probe:
> > >> > >
> > >> > > drivers/dma/ioat/pci.c:
> > >> > >                 err = ioat3_dma_probe(device, ioat_dca_enabled);
> > >> > >         else
> > >> > >                 return -ENODEV;
> > >> > >
> > >> > >         if (err) {
> > >> > >                 dev_err(dev, "Intel(R) I/OAT DMA Engine init
> > >> > > failed\n");
> > >> > >                 return -ENODEV;
> > >> > --
> > >> > ~Vinod

-- 
~Vinod

[toc] | [prev] | [next] | [standalone]

#1403950

From"Jiang, Dave" <dave.jiang@intel.com>
Date2016-05-19 22:20 +0200
Message-ID<rACAh-4uH-9@gated-at.bofh.it>
In reply to#1403874

> -----Original Message-----
> From: Koul, Vinod
> Sent: Thursday, May 19, 2016 10:23 AM
> To: Jiang, Dave <dave.jiang@intel.com>
> Cc: Gavin Guo <gavin.guo@canonical.com>; dmaengine@vger.kernel.org; linux-kernel@vger.kernel.org; Williams, Dan J
> <dan.j.williams@intel.com>
> Subject: Re: ioatdma(Intel(R) I/OAT DMA Engine init failed)
> 
> On Thu, May 19, 2016 at 08:19:30PM +0530, Jiang, Dave wrote:
> > > -----Original Message-----
> > > From: Gavin Guo [mailto:gavin.guo@canonical.com]
> > > Sent: Wednesday, May 18, 2016 8:19 PM
> > > To: Jiang, Dave <dave.jiang@intel.com>
> > > Cc: Koul, Vinod <vinod.koul@intel.com>; dmaengine@vger.kernel.org; linux-kernel@vger.kernel.org; Williams, Dan J
> > > <dan.j.williams@intel.com>
> > > Subject: Re: ioatdma(Intel(R) I/OAT DMA Engine init failed)
> > >
> > > On Thu, May 19, 2016 at 12:49 AM, Jiang, Dave <dave.jiang@intel.com> wrote:
> > > > On Wed, 2016-05-18 at 13:27 +0000, Gavin Guo wrote:
> > > >> On Tue, May 17, 2016 at 6:06 PM, Vinod Koul <vinod.koul@intel.com>
> > > >> wrote:
> > > >> >
> > > >> > On Mon, May 16, 2016 at 06:08:20PM +0800, Gavin Guo wrote:
> > > >> > >
> > > >> > > The following error messages can be observed on the Intel
> > > >> > > Haswell-E
> > > >> > > chipset with v3.13 kernel. After the analysis, I found there is
> > > >> > > no
> > > >> > > difference in the logic of these error messages in the current
> > > >> > > upstream kernel. I also searched the git log and can't find any
> > > >> > > commit
> > > >> > > which is fix to the error(correct me if I am wrong). The
> > > >> > > following is
> > > >> > > the detail, and I'll really appreciate if there is any comment.
> > > >> > > :)
> > > >> > 3.13 is ancient, can you check this on latest kernel
> > > >> Thank you for the comment. It's running on the production system.
> > > >> However,
> > > >> I'll try to figure out if it's possible to test the latest kernel.
> > > >
> > > > I wonder if you don't have the extended PCI config space access enabled
> > > > in your kernel config.
> > >
> > > Really thanks for your advice. :)
> > >
> > > I searched the internet about the extended PCI config space and found
> > > the link:
> > >
> > > [Patch v2] Make PCI extended config space (MMCONFIG) a driver opt-in
> > > http://lwn.net/Articles/263288/
> >
> > Can you try calling pci_enable_ext_config() in the PCI probe for your kernel? I just haven't seen this issue in the latest kernel.
> 
> Do we need that to be called explicitly by driver, should that not be enabled
> by default?

It used to be but the patch seems to indicate it's an opt in thing. But what I don't get is I have never encountered this issue. 

> 
> >
> > >
> > > And I checked the config and found the CONFIG_PCI_MMCONFIG=y. The
> > > following string also can be observed in the dmesg:
> > >
> > > [    1.419853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at
> > > [mem0x80000000-0x8fffffff] (base 0x80000000)
> > > [    1.419855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
> > >
> > > It seems the extended PCI config space is enabled. If there is
> > > anything missed?
> > >
> > > >
> > > >>
> > > >> >
> > > >> >
> > > >> > >
> > > >> > >
> > > >> > > ioatdma 0000:00:04.0: channel error register unreachable
> > > >> > > ioatdma 0000:00:04.0: channel enumeration error
> > > >> > > ioatdma 0000:00:04.0: Intel(R) I/OAT DMA Engine init failed
> > > >> > > ioatdma 0000:00:04.1: channel error register unreachable
> > > >> > > ioatdma 0000:00:04.1: channel enumeration error
> > > >> > > ioatdma 0000:00:04.1: Intel(R) I/OAT DMA Engine init failed
> > > >> > > ...
> > > >> > > ioatdma 0000:00:04.7: channel error register unreachable
> > > >> > > ioatdma 0000:00:04.7: channel enumeration error
> > > >> > > ioatdma 0000:00:04.7: Intel(R) I/OAT DMA Engine init failed
> > > >> > > mei_me 0000:00:16.0: initialization failed.
> > > >> > >
> > > >> > > There are 8 I/OAT DMA controllers on the Haswell-E chipset:
> > > >> > > 8086:2f20 ~ 8086:2f27
> > > >> > > 80:04.0 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 0 (rev 02)
> > > >> > > 80:04.1 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 1 (rev 02)
> > > >> > > 80:04.2 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 2 (rev 02)
> > > >> > > 80:04.3 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 3 (rev 02)
> > > >> > > 80:04.4 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 4 (rev 02)
> > > >> > > 80:04.5 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 5 (rev 02)
> > > >> > > 80:04.6 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 6 (rev 02)
> > > >> > > 80:04.7 System peripheral: Intel Corporation Haswell-E DMA
> > > >> > > Channel 7 (rev 02)
> > > >> > >
> > > >> > > Analysis:
> > > >> > > The bug happens when the driver is resetting DMA controller, this
> > > >> > > is
> > > >> > > the sequence: The function, ioat_pci_probe, is called when the
> > > >> > > DMA
> > > >> > > controller is detected by the PCI bus. Then,
> > > >> > > ioat3_dma_probe -> ioat_probe -> ioat2_enumerate_channels ->
> > > >> > > ioat3_reset_hw. The following code can be found in the
> > > >> > > ioat3_reset_hw:
> > > >> > >
> > > >> > > drivers/dma/ioat/dma_v3.c:
> > > >> > >         chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
> > > >> > >         writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
> > > >> > > ...
> > > >> > >         err = pci_read_config_dword(pdev,
> > > >> > > IOAT_PCI_CHANERR_INT_OFFSET, &chanerr);
> > > >> > > if (err) {
> > > >> > > dev_err(&pdev->dev,
> > > >> > > "channel error register unreachable\n");
> > > >> > > return err;
> > > >> > > }
> > > >> > >
> > > >> > > Obviously, there are something wrong in the channel error
> > > >> > > register
> > > >> > > reset process. Then all the way back to ioat_probe(). Because the
> > > >> > > error happens, the dma->chancnt will be set to 0:
> > > >> > >
> > > >> > > drivers/dma/ioat/dma.c:
> > > >> > >         if (!dma->chancnt) {
> > > >> > >                 dev_err(dev, "channel enumeration error\n");
> > > >> > >                 goto err_setup_interrupts;
> > > >> > >         }
> > > >> > >
> > > >> > > Finally back to ioat_pci_probe:
> > > >> > >
> > > >> > > drivers/dma/ioat/pci.c:
> > > >> > >                 err = ioat3_dma_probe(device, ioat_dca_enabled);
> > > >> > >         else
> > > >> > >                 return -ENODEV;
> > > >> > >
> > > >> > >         if (err) {
> > > >> > >                 dev_err(dev, "Intel(R) I/OAT DMA Engine init
> > > >> > > failed\n");
> > > >> > >                 return -ENODEV;
> > > >> > --
> > > >> > ~Vinod
> 
> --
> ~Vinod

[toc] | [prev] | [next] | [standalone]

#1403982

FromYinghai Lu <yinghai@kernel.org>
Date2016-05-20 00:20 +0200
Message-ID<rAEsp-5GL-9@gated-at.bofh.it>
In reply to#1403950
On Thu, May 19, 2016 at 1:17 PM, Jiang, Dave <dave.jiang@intel.com> wrote:
>> > > And I checked the config and found the CONFIG_PCI_MMCONFIG=y. The
>> > > following string also can be observed in the dmesg:
>> > >
>> > > [    1.419853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at
>> > > [mem0x80000000-0x8fffffff] (base 0x80000000)
>> > > [    1.419855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
>> > >
>> > > It seems the extended PCI config space is enabled. If there is
>> > > anything missed?

how about output for  "lspci -vvxxxx" ?

[toc] | [prev] | [next] | [standalone]

#1406684

FromGavin Guo <gavin.guo@canonical.com>
Date2016-05-25 09:00 +0200
Message-ID<rCAXo-6au-3@gated-at.bofh.it>
In reply to#1403982
On Fri, May 20, 2016 at 6:17 AM, Yinghai Lu <yinghai@kernel.org> wrote:
> On Thu, May 19, 2016 at 1:17 PM, Jiang, Dave <dave.jiang@intel.com> wrote:
>>> > > And I checked the config and found the CONFIG_PCI_MMCONFIG=y. The
>>> > > following string also can be observed in the dmesg:
>>> > >
>>> > > [    1.419853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at
>>> > > [mem0x80000000-0x8fffffff] (base 0x80000000)
>>> > > [    1.419855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
>>> > >
>>> > > It seems the extended PCI config space is enabled. If there is
>>> > > anything missed?
>
> how about output for  "lspci -vvxxxx" ?

Sorry that my client is slow in response. I'll bring it up if I
receive any information.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web