Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1620770 > unrolled thread

Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without struct page

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without  struct page Shuah Khan <shuahkh@osg.samsung.com> - 2017-04-11 01:00 +0200
    Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without  struct page Marek Szyprowski <m.szyprowski@samsung.com> - 2017-04-14 10:00 +0200
      Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without  struct page Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-04-14 11:50 +0200
        Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without  struct page Shuah Khan <shuahkh@osg.samsung.com> - 2017-04-17 03:20 +0200
          Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without  struct page Russell King - ARM Linux <linux@armlinux.org.uk> - 2017-04-17 12:30 +0200
        Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without  struct page Shuah Khan <shuahkh@osg.samsung.com> - 2017-04-20 01:40 +0200

#1620770 — Re: [PATCH] arm: dma: fix sharing of coherent DMA memory without struct page

On 04/06/2017 06:01 AM, Marek Szyprowski wrote:
> Hi Shuah,
> 
> On 2017-04-05 18:02, Shuah Khan wrote:
>> When coherent DMA memory without struct page is shared, importer
>> fails to find the page and runs into kernel page fault when it
>> tries to dmabuf_ops_attach/map_sg/map_page the invalid page found
>> in the sg_table. Please see www.spinics.net/lists/stable/msg164204.html
>> for more information on this problem.
>>
>> This solution allows coherent DMA memory without struct page to be
>> shared by providing a way for the exporter to tag the DMA buffer as
>> a special buffer without struct page association and passing the
>> information in sg_table to the importer. This information is used
>> in attach/map_sg to avoid cleaning D-cache and mapping.
>>
>> The details of the change are:
>>
>> Framework:
>> - Add a new dma_attrs field to struct scatterlist.
>> - Add a new DMA_ATTR_DEV_COHERENT_NOPAGE attribute to clearly identify
>>    Coherent memory without struct page.
>> - Add a new dma_check_dev_coherent() interface to check if memory is
>>    the device coherent area. There is no way to tell where the memory
>>    returned by dma_alloc_attrs() came from.
>>
>> Exporter logic:
>> - Add logic to vb2_dc_alloc() to call dma_check_dev_coherent() and set
>>    DMA_ATTR_DEV_COHERENT_NOPAGE based the results of the check. This is
>>    done in the exporter context.
>> - Add logic to arm_dma_get_sgtable() to identify memory without struct
>>    page using DMA_ATTR_DEV_COHERENT_NOPAGE attribute. If this attr is
>>    set, arm_dma_get_sgtable() will set page as the cpu_addr and update
>>    dma_address and dma_attrs fields in struct scatterlist for this sgl.
>>    This is done in exporter context when buffer is exported. With this
>>    Note: This change is made on top of Russell King's patch that added
>>    !pfn_valid(pfn) check to arm_dma_get_sgtable() to error out on invalid
>>    pages. Coherent memory without struct page will trigger this error.
>>
>> Importer logic:
>> - Add logic to vb2_dc_dmabuf_ops_attach() to identify memory without
>>    struct page using DMA_ATTR_DEV_COHERENT_NOPAGE attribute when it copies
>>    the sg_table from the exporter. It will copy dma_attrs and dma_address
>>    fields. With this logic, dmabuf_ops_attach will no longer trip on an
>>    invalid page.
>> - Add logic to arm_dma_map_sg() to avoid mapping the page when sg_table
>>    has DMA_ATTR_DEV_COHERENT_NOPAGE buffer.
>> - Add logic to arm_dma_unmap_sg() to do nothing for sg entries with
>>    DMA_ATTR_DEV_COHERENT_NOPAGE attribute.
>>
>> Without this change the following use-case that runs into kernel
>> pagefault when importer tries to attach the exported buffer.
>>
>> With this change it works: (what a relief after watching pagefaults for
>> weeks!!)
>>
>> gst-launch-1.0 filesrc location=~/GH3_MOV_HD.mp4 ! qtdemux ! h264parse ! v4l2video4dec capture-io-mode=dmabuf ! v4l2video7convert output-io-mode=dmabuf-import ! kmssink force-modesetting=true
>>
>> I am sending RFC patch to get feedback on the approach and see if I missed
>> anything.
> 
> Frankly, once You decided to hack around dma-buf and issues with coherent,
> carved out memory, it might be a bit better to find the ultimate solution
> instead of the another hack. Please note that it will still not allow to
> share a buffer allocated from carved-out memory and a device, which is
> behind IOMMU.

With your patch s5p-mfc patch series does address the problem for this
use-case for 4.12 onwards. However I am still concerned about prior
release and this pagefault is bad.

Invalid page test partially solves the problem. Would it helpful to
at least prevent the pagfault with a definitive test. Please see my
response to Russell. Let me know your thoughts on that.

> 
> I thought a bit about this and the current shape of dma-buf code.
> 
> IMHO the proper way of solving all those issues would be to replace
> dma-buf internal representation of the memory from struct scatter_list
> to pfn array. This would really solve the problem of buffers which
> cannot be properly represented by scatter lists/struct pages and would
> even allow sharing buffers between all kinds of devices. Scatter-lists
> are also quite over-engineered structures to represent a single buffer
> (pfn array is a bit more compact representation). Also there is a lots
> of buggy code which use scatter-list in a bit creative way (like
> assuming that each page maps to a single scatter list entry for
> example). The only missing piece, required for such change would be
> extending DMA-mapping with dma_map_pfn() interface.

I agree with you on scatterlists being clumsy. Changing over to pfn array
could simplify things. I am exploring a slightly different option that
might not require too many changes. I will respond with concrete ideas
later on this week.

> 
> This would be however quite large task, especially taking into account
> all current users of DMA-buf framework...

Yeah it will be a large task.

thanks,
-- Shuah

> 
>> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
>> ---
>>   arch/arm/mm/dma-mapping.c                      | 34 ++++++++++++++++++++++----
>>   drivers/base/dma-coherent.c                    | 25 +++++++++++++++++++
>>   drivers/media/v4l2-core/videobuf2-dma-contig.c |  6 +++++
>>   include/linux/dma-mapping.h                    |  8 ++++++
>>   include/linux/scatterlist.h                    |  1 +
>>   5 files changed, 69 insertions(+), 5 deletions(-)
>> [...]
> 
> Best regards

[toc] | [next] | [standalone]

#1623551

Hi Shuah,

On 2017-04-11 00:50, Shuah Khan wrote:
> On 04/06/2017 06:01 AM, Marek Szyprowski wrote:
>> On 2017-04-05 18:02, Shuah Khan wrote:
>>> When coherent DMA memory without struct page is shared, importer
>>> fails to find the page and runs into kernel page fault when it
>>> tries to dmabuf_ops_attach/map_sg/map_page the invalid page found
>>> in the sg_table. Please see www.spinics.net/lists/stable/msg164204.html
>>> for more information on this problem.
>>>
>>> This solution allows coherent DMA memory without struct page to be
>>> shared by providing a way for the exporter to tag the DMA buffer as
>>> a special buffer without struct page association and passing the
>>> information in sg_table to the importer. This information is used
>>> in attach/map_sg to avoid cleaning D-cache and mapping.
>>>
>>> The details of the change are:
>>>
>>> Framework:
>>> - Add a new dma_attrs field to struct scatterlist.
>>> - Add a new DMA_ATTR_DEV_COHERENT_NOPAGE attribute to clearly identify
>>>     Coherent memory without struct page.
>>> - Add a new dma_check_dev_coherent() interface to check if memory is
>>>     the device coherent area. There is no way to tell where the memory
>>>     returned by dma_alloc_attrs() came from.
>>>
>>> Exporter logic:
>>> - Add logic to vb2_dc_alloc() to call dma_check_dev_coherent() and set
>>>     DMA_ATTR_DEV_COHERENT_NOPAGE based the results of the check. This is
>>>     done in the exporter context.
>>> - Add logic to arm_dma_get_sgtable() to identify memory without struct
>>>     page using DMA_ATTR_DEV_COHERENT_NOPAGE attribute. If this attr is
>>>     set, arm_dma_get_sgtable() will set page as the cpu_addr and update
>>>     dma_address and dma_attrs fields in struct scatterlist for this sgl.
>>>     This is done in exporter context when buffer is exported. With this
>>>     Note: This change is made on top of Russell King's patch that added
>>>     !pfn_valid(pfn) check to arm_dma_get_sgtable() to error out on invalid
>>>     pages. Coherent memory without struct page will trigger this error.
>>>
>>> Importer logic:
>>> - Add logic to vb2_dc_dmabuf_ops_attach() to identify memory without
>>>     struct page using DMA_ATTR_DEV_COHERENT_NOPAGE attribute when it copies
>>>     the sg_table from the exporter. It will copy dma_attrs and dma_address
>>>     fields. With this logic, dmabuf_ops_attach will no longer trip on an
>>>     invalid page.
>>> - Add logic to arm_dma_map_sg() to avoid mapping the page when sg_table
>>>     has DMA_ATTR_DEV_COHERENT_NOPAGE buffer.
>>> - Add logic to arm_dma_unmap_sg() to do nothing for sg entries with
>>>     DMA_ATTR_DEV_COHERENT_NOPAGE attribute.
>>>
>>> Without this change the following use-case that runs into kernel
>>> pagefault when importer tries to attach the exported buffer.
>>>
>>> With this change it works: (what a relief after watching pagefaults for
>>> weeks!!)
>>>
>>> gst-launch-1.0 filesrc location=~/GH3_MOV_HD.mp4 ! qtdemux ! h264parse ! v4l2video4dec capture-io-mode=dmabuf ! v4l2video7convert output-io-mode=dmabuf-import ! kmssink force-modesetting=true
>>>
>>> I am sending RFC patch to get feedback on the approach and see if I missed
>>> anything.
>> Frankly, once You decided to hack around dma-buf and issues with coherent,
>> carved out memory, it might be a bit better to find the ultimate solution
>> instead of the another hack. Please note that it will still not allow to
>> share a buffer allocated from carved-out memory and a device, which is
>> behind IOMMU.
> With your patch s5p-mfc patch series does address the problem for this
> use-case for 4.12 onwards. However I am still concerned about prior
> release and this pagefault is bad.

Right. It should simply fail with error code instead of pagefault.

> Invalid page test partially solves the problem. Would it helpful to
> at least prevent the pagfault with a definitive test. Please see my
> response to Russell. Let me know your thoughts on that.
>
>> I thought a bit about this and the current shape of dma-buf code.
>>
>> IMHO the proper way of solving all those issues would be to replace
>> dma-buf internal representation of the memory from struct scatter_list
>> to pfn array. This would really solve the problem of buffers which
>> cannot be properly represented by scatter lists/struct pages and would
>> even allow sharing buffers between all kinds of devices. Scatter-lists
>> are also quite over-engineered structures to represent a single buffer
>> (pfn array is a bit more compact representation). Also there is a lots
>> of buggy code which use scatter-list in a bit creative way (like
>> assuming that each page maps to a single scatter list entry for
>> example). The only missing piece, required for such change would be
>> extending DMA-mapping with dma_map_pfn() interface.
> I agree with you on scatterlists being clumsy. Changing over to pfn array
> could simplify things. I am exploring a slightly different option that
> might not require too many changes. I will respond with concrete ideas
> later on this week.

It looks that a similar issue is being worked on, see the following thread:
https://lkml.org/lkml/2017/4/13/710

>> This would be however quite large task, especially taking into account
>> all current users of DMA-buf framework...
> Yeah it will be a large task.

Maybe once scatterlist are switched to pfns, changing dmabuf internal
memory representation to pfn array might be much easier.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

[toc] | [prev] | [next] | [standalone]

#1623619

On Fri, Apr 14, 2017 at 09:56:07AM +0200, Marek Szyprowski wrote:
> >>This would be however quite large task, especially taking into account
> >>all current users of DMA-buf framework...
> >Yeah it will be a large task.
> 
> Maybe once scatterlist are switched to pfns, changing dmabuf internal
> memory representation to pfn array might be much easier.

Switching to a PFN array won't work either as we have no cross-arch
way to translate PFNs to a DMA address and vice versa.  Yes, we have
them in ARM, but they are an _implementation detail_ of ARM's
DMA API support, they are not for use by drivers.

So, the very first problem that needs solving is this:

  How do we go from a coherent DMA allocation for device X to a set
  of DMA addresses for device Y.

Essentially, we need a way of remapping the DMA buffer for use with
another device, and returning a DMA address suitable for that device.
This could well mean that we need to deal with setting up an IOMMU
mapping.  My guess is that this needs to happen at the DMA coherent
API level - the DMA coherent API needs to be augmented with support
for this.  I'll call this "DMA coherent remap".

We then need to think about how to pass this through the dma-buf API.
dma_map_sg() is done by the exporter, who should know what kind of
memory is being exported.  The exporter can avoid calling dma_map_sg()
if it knows in advance that it is exporting DMA coherent memory.
Instead, the exporter can simply create a scatterlist with the DMA
address and DMA length prepopulated with the results of the DMA
coherent remap operation above.

What the scatterlist can't carry in this case is a set of valid
struct page pointers, and an importer must not walk the scatterlist
expecting to get at the virtual address parameters or struct page
pointers.

On the mmap() side of things, remember that DMA coherent allocations
may require special mapping into userspace, and which can only be
mapped by the DMA coherent mmap support.  kmap etc will also need to
be different.  So it probably makes sense for DMA coherent dma-buf
exports to use a completely separate set of dma_buf_ops from the
streaming version.

I think this is the easiest approach to solving the problem without
needing massive driver changes all over the kernel.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1624493

On 04/14/2017 03:46 AM, Russell King - ARM Linux wrote:
> On Fri, Apr 14, 2017 at 09:56:07AM +0200, Marek Szyprowski wrote:
>>>> This would be however quite large task, especially taking into account
>>>> all current users of DMA-buf framework...
>>> Yeah it will be a large task.
>>
>> Maybe once scatterlist are switched to pfns, changing dmabuf internal
>> memory representation to pfn array might be much easier.
> 
> Switching to a PFN array won't work either as we have no cross-arch
> way to translate PFNs to a DMA address and vice versa.  Yes, we have
> them in ARM, but they are an _implementation detail_ of ARM's
> DMA API support, they are not for use by drivers.
> 
> So, the very first problem that needs solving is this:
> 
>   How do we go from a coherent DMA allocation for device X to a set
>   of DMA addresses for device Y.
> 
> Essentially, we need a way of remapping the DMA buffer for use with
> another device, and returning a DMA address suitable for that device.
> This could well mean that we need to deal with setting up an IOMMU
> mapping.  My guess is that this needs to happen at the DMA coherent
> API level - the DMA coherent API needs to be augmented with support
> for this.  I'll call this "DMA coherent remap".
> 
> We then need to think about how to pass this through the dma-buf API.
> dma_map_sg() is done by the exporter, who should know what kind of
> memory is being exported.  The exporter can avoid calling dma_map_sg()
> if it knows in advance that it is exporting DMA coherent memory.
> Instead, the exporter can simply create a scatterlist with the DMA
> address and DMA length prepopulated with the results of the DMA
> coherent remap operation above.

The only way to conclusively say that it is coming from coherent area
is at the time it is getting allocated in dma_alloc_from_coherent().
Since dma_alloc_attrs() will go on to find memory from other areas if
dma_alloc_from_coherent() doesn't allocate memory.

dma_get_sgtable_attrs() is what is used by the exporter to create the
sg_table. One way to do this cleanly without needing to check buffer
type flags would be to add a set of sg_table ops: get_sgtable,
map_sg, and unmap_sg. Sounds like sg_table interfaces need to be in
dma_buf_ops level. More below.

> 
> What the scatterlist can't carry in this case is a set of valid
> struct page pointers, and an importer must not walk the scatterlist
> expecting to get at the virtual address parameters or struct page
> pointers.
> 
> On the mmap() side of things, remember that DMA coherent allocations
> may require special mapping into userspace, and which can only be
> mapped by the DMA coherent mmap support.  kmap etc will also need to
> be different.  So it probably makes sense for DMA coherent dma-buf
> exports to use a completely separate set of dma_buf_ops from the
> streaming version.

How about adding get_sgtable, map_sg, unmap_sg to dma_buf_ops. The right
ops need to be installed based on buffer type. As I mentioned before, we
don't know which memory we got until dma_alloc_from_coherent() finds
memory in dev->mem area. So how about using the dma_check_dev_coherent()
to determine which ops we need. These could be set based on buffer type.
vb2_dc_get_dmabuf() can do that.

I think this will work.

thanks,
-- Shuah

[toc] | [prev] | [next] | [standalone]

#1624627

On Sun, Apr 16, 2017 at 07:10:21PM -0600, Shuah Khan wrote:
> On 04/14/2017 03:46 AM, Russell King - ARM Linux wrote:
> > On Fri, Apr 14, 2017 at 09:56:07AM +0200, Marek Szyprowski wrote:
> >>>> This would be however quite large task, especially taking into account
> >>>> all current users of DMA-buf framework...
> >>> Yeah it will be a large task.
> >>
> >> Maybe once scatterlist are switched to pfns, changing dmabuf internal
> >> memory representation to pfn array might be much easier.
> > 
> > Switching to a PFN array won't work either as we have no cross-arch
> > way to translate PFNs to a DMA address and vice versa.  Yes, we have
> > them in ARM, but they are an _implementation detail_ of ARM's
> > DMA API support, they are not for use by drivers.
> > 
> > So, the very first problem that needs solving is this:
> > 
> >   How do we go from a coherent DMA allocation for device X to a set
> >   of DMA addresses for device Y.
> > 
> > Essentially, we need a way of remapping the DMA buffer for use with
> > another device, and returning a DMA address suitable for that device.
> > This could well mean that we need to deal with setting up an IOMMU
> > mapping.  My guess is that this needs to happen at the DMA coherent
> > API level - the DMA coherent API needs to be augmented with support
> > for this.  I'll call this "DMA coherent remap".
> > 
> > We then need to think about how to pass this through the dma-buf API.
> > dma_map_sg() is done by the exporter, who should know what kind of
> > memory is being exported.  The exporter can avoid calling dma_map_sg()
> > if it knows in advance that it is exporting DMA coherent memory.
> > Instead, the exporter can simply create a scatterlist with the DMA
> > address and DMA length prepopulated with the results of the DMA
> > coherent remap operation above.
> 
> The only way to conclusively say that it is coming from coherent area
> is at the time it is getting allocated in dma_alloc_from_coherent().
> Since dma_alloc_attrs() will go on to find memory from other areas if
> dma_alloc_from_coherent() doesn't allocate memory.

Sorry, I disagree.

The only thing that matters is "did this memory come from
dma_alloc_coherent()".  It doesn't matter where dma_alloc_coherent()
ultimately got the memory from, it's memory from the coherent allocator
interface, and it should not be passed back into the streaming APIs.
It is, after all, DMA _coherent_ memory, passing it into the streaming
APIs which is for DMA _noncoherent_ memory is insane - the streaming APIs
can bring extra expensive cache flushes, which are not required for
DMA _coherent_ memory.

The exporter should know where it got the memory from.  It's really not
sane for anyone except the _original_ allocator to be exporting memory
through a DMA buffer - only the original allocator knows the properties
of that memory, and how to map it, whether that be for DMA, kmap or
mmap.

If a dmabuf is imported into a driver and then re-exported, the original
dmabuf should be what is re-exported, not some creation of the driver -
the re-exporting driver can't know what the properties of the memory
backing the dmabuf are, so anything else is just insane.

> How about adding get_sgtable, map_sg, unmap_sg to dma_buf_ops. The right
> ops need to be installed based on buffer type. As I mentioned before, we
> don't know which memory we got until dma_alloc_from_coherent() finds
> memory in dev->mem area. So how about using the dma_check_dev_coherent()
> to determine which ops we need. These could be set based on buffer type.
> vb2_dc_get_dmabuf() can do that.

Given my statement above, I don't believe any of that is necessary.  All
memory allocated from dma_alloc_coherent() is DMA coherent.  So, if
memory was obtained from dma_alloc_coherent() or similar, then it must
not be passed to the streaming DMA API.  It doesn't matter where it
ultimately came from.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[toc] | [prev] | [next] | [standalone]

#1626878

Hi Russell, and Marek,

On 04/14/2017 03:46 AM, Russell King - ARM Linux wrote:
> On Fri, Apr 14, 2017 at 09:56:07AM +0200, Marek Szyprowski wrote:
>>>> This would be however quite large task, especially taking into account
>>>> all current users of DMA-buf framework...
>>> Yeah it will be a large task.
>>
>> Maybe once scatterlist are switched to pfns, changing dmabuf internal
>> memory representation to pfn array might be much easier.
> 
> Switching to a PFN array won't work either as we have no cross-arch
> way to translate PFNs to a DMA address and vice versa.  Yes, we have
> them in ARM, but they are an _implementation detail_ of ARM's
> DMA API support, they are not for use by drivers.
> 
> So, the very first problem that needs solving is this:
> 
>   How do we go from a coherent DMA allocation for device X to a set
>   of DMA addresses for device Y.
> 
> Essentially, we need a way of remapping the DMA buffer for use with
> another device, and returning a DMA address suitable for that device.
> This could well mean that we need to deal with setting up an IOMMU
> mapping.  My guess is that this needs to happen at the DMA coherent
> API level - the DMA coherent API needs to be augmented with support
> for this.  I'll call this "DMA coherent remap".
> 
> We then need to think about how to pass this through the dma-buf API.
> dma_map_sg() is done by the exporter, who should know what kind of
> memory is being exported.  The exporter can avoid calling dma_map_sg()
> if it knows in advance that it is exporting DMA coherent memory.
> Instead, the exporter can simply create a scatterlist with the DMA
> address and DMA length prepopulated with the results of the DMA
> coherent remap operation above.

As Russell pointed to armama-drm case, I looked at that closely.
armada-drm is creating sg_table and populating it with DMA-address in
its map_dma_buf ops and unmap_dma_buf ops handles the special case and
doesn't call dma_unmap_sg().

In the case of drm, gem_prime_map_dma_buf interfaces and the common
drm_gem_map_dma_buf() will need modification to not do dma_map_sg()
and create scatterlist with the DMA address and DMA length instead.
We have to get drm_gem_map_dma_buf() info. to have it not do dma_map_sg()
and create scatterlist.

Focusing on drm for now, looks like there are probably about 15 or so
map_dma_buf interfaces will need to handle coherent memory case.

> 
> What the scatterlist can't carry in this case is a set of valid
> struct page pointers, and an importer must not walk the scatterlist
> expecting to get at the virtual address parameters or struct page
> pointers.

Right - importers need handling to not walk the sg_list and handle
it differently. Is there a good example drm you can point me to for
this? aramda-drm seems to special case this in armada_gem_map_import()
if I am not mistaken.

> 
> On the mmap() side of things, remember that DMA coherent allocations
> may require special mapping into userspace, and which can only be
> mapped by the DMA coherent mmap support.  kmap etc will also need to
> be different.  So it probably makes sense for DMA coherent dma-buf
> exports to use a completely separate set of dma_buf_ops from the
> streaming version.
> 

I agree. It would make is easier and also limits the scope of changes.

> I think this is the easiest approach to solving the problem without
> needing massive driver changes all over the kernel.
> 

Anyway this is a quick note to say that I am looking into this and
haven't drooped it :)

thanks,
-- Shuah

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web