Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1348050 > unrolled thread

Suspicious error for CMA stress test

Back to article view | Back to linux.kernel

Contents

  Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-02 15:00 +0100
    Re: Suspicious error for CMA stress test Laura Abbott <labbott@redhat.com> - 2016-03-03 02:30 +0100
      Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-03 07:30 +0100
      Re: Suspicious error for CMA stress test Joonsoo Kim <js1304@gmail.com> - 2016-03-03 08:50 +0100
        Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-03 09:00 +0100
        Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-03 14:00 +0100
          Re: Suspicious error for CMA stress test Laura Abbott <labbott@redhat.com> - 2016-03-03 20:00 +0100
            Re: Suspicious error for CMA stress test Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-03-04 03:10 +0100
              Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-04 07:20 +0100
          Re: Suspicious error for CMA stress test Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-03-04 03:10 +0100
            Re: Suspicious error for CMA stress test Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-03-04 05:40 +0100
              Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-04 07:10 +0100
                Re: Suspicious error for CMA stress test Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-03-04 07:40 +0100
                  Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-04 08:40 +0100
            Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-04 06:40 +0100
            Re: Suspicious error for CMA stress test Hanjun Guo <guohanjun@huawei.com> - 2016-03-04 08:10 +0100

#1348050 — Suspicious error for CMA stress test

Hi,

I came across a suspicious error for CMA stress test:

Before the test, I got:
-bash-4.3# cat /proc/meminfo | grep Cma
CmaTotal:         204800 kB
CmaFree:          195044 kB


After running the test:
-bash-4.3# cat /proc/meminfo | grep Cma
CmaTotal:         204800 kB
CmaFree:         6602584 kB

So the freed CMA memory is more than total..

Also the the MemFree is more than mem total:

-bash-4.3# cat /proc/meminfo          
MemTotal:       16342016 kB
MemFree:        22367268 kB
MemAvailable:   22370528 kB

Here is the kernel module doing the stress test below (if the test case
is wrong, correct me), any help would be great appreciated.

The test is running on ARM64 platform (hisilicon D02) with 4.4 kernel, I think
the 4.5-rc is the same as I didn't notice the updates for it.

int malloc_dma(void *data)
{
    void *vaddr;
    struct platform_device * pdev=(struct platform_device*)data;
    dma_addr_t dma_handle;
    int i;

    for(i=0; i<1000; i++) {
        vaddr=dma_alloc_coherent(&pdev->dev, malloc_size, &dma_handle, GFP_KERNEL);
        if (!vaddr)
            pr_err("alloc cma memory failed!\n");

        mdelay(1);

        if (vaddr)
                dma_free_coherent(&pdev->dev,malloc_size,vaddr, dma_handle);
    }
    pr_info("alloc free cma memory success return!\n");
    return 0;
}

static int dma_alloc_coherent_init(struct platform_device *pdev)
{
    int i;

    for(i=0; i<100; i++)   {
        task[i] = kthread_create(malloc_dma,pdev,"malloc_dma_%d",i);
        if(!task[i]) {
            printk("kthread_create faile %d\n",i);
            continue;
        }
        wake_up_process(task[i]);
    }
    return 0;
}

Thanks
Hanjun

The whole /proc/meminfo:

-bash-4.3# cat /proc/meminfo          
MemTotal:       16342016 kB
MemFree:        22367268 kB
MemAvailable:   22370528 kB
Buffers:            4292 kB
Cached:            36444 kB
SwapCached:            0 kB
Active:            23564 kB
Inactive:          25360 kB
Active(anon):       8424 kB
Inactive(anon):       64 kB
Active(file):      15140 kB
Inactive(file):    25296 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          8196 kB
Mapped:            16448 kB
Shmem:               296 kB
Slab:              26832 kB
SReclaimable:       6300 kB
SUnreclaim:        20532 kB
KernelStack:        3088 kB
PageTables:          404 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8171008 kB
Committed_AS:      34336 kB
VmallocTotal:   258998208 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:         0 kB
CmaTotal:         204800 kB
CmaFree:         6602584 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

[toc] | [next] | [standalone]

#1348669

(cc -mm and Joonsoo Kim)

On 03/02/2016 05:52 AM, Hanjun Guo wrote:
> Hi,
>
> I came across a suspicious error for CMA stress test:
>
> Before the test, I got:
> -bash-4.3# cat /proc/meminfo | grep Cma
> CmaTotal:         204800 kB
> CmaFree:          195044 kB
>
>
> After running the test:
> -bash-4.3# cat /proc/meminfo | grep Cma
> CmaTotal:         204800 kB
> CmaFree:         6602584 kB
>
> So the freed CMA memory is more than total..
>
> Also the the MemFree is more than mem total:
>
> -bash-4.3# cat /proc/meminfo
> MemTotal:       16342016 kB
> MemFree:        22367268 kB
> MemAvailable:   22370528 kB
>
> Here is the kernel module doing the stress test below (if the test case
> is wrong, correct me), any help would be great appreciated.
>
> The test is running on ARM64 platform (hisilicon D02) with 4.4 kernel, I think
> the 4.5-rc is the same as I didn't notice the updates for it.
>
> int malloc_dma(void *data)
> {
>      void *vaddr;
>      struct platform_device * pdev=(struct platform_device*)data;
>      dma_addr_t dma_handle;
>      int i;
>
>      for(i=0; i<1000; i++) {
>          vaddr=dma_alloc_coherent(&pdev->dev, malloc_size, &dma_handle, GFP_KERNEL);
>          if (!vaddr)
>              pr_err("alloc cma memory failed!\n");
>
>          mdelay(1);
>
>          if (vaddr)
>                  dma_free_coherent(&pdev->dev,malloc_size,vaddr, dma_handle);
>      }
>      pr_info("alloc free cma memory success return!\n");
>      return 0;
> }
>
> static int dma_alloc_coherent_init(struct platform_device *pdev)
> {
>      int i;
>
>      for(i=0; i<100; i++)   {
>          task[i] = kthread_create(malloc_dma,pdev,"malloc_dma_%d",i);
>          if(!task[i]) {
>              printk("kthread_create faile %d\n",i);
>              continue;
>          }
>          wake_up_process(task[i]);
>      }
>      return 0;
> }
>
> Thanks
> Hanjun
>
> The whole /proc/meminfo:
>
> -bash-4.3# cat /proc/meminfo
> MemTotal:       16342016 kB
> MemFree:        22367268 kB
> MemAvailable:   22370528 kB
> Buffers:            4292 kB
> Cached:            36444 kB
> SwapCached:            0 kB
> Active:            23564 kB
> Inactive:          25360 kB
> Active(anon):       8424 kB
> Inactive(anon):       64 kB
> Active(file):      15140 kB
> Inactive(file):    25296 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> SwapTotal:             0 kB
> SwapFree:              0 kB
> Dirty:                 0 kB
> Writeback:             0 kB
> AnonPages:          8196 kB
> Mapped:            16448 kB
> Shmem:               296 kB
> Slab:              26832 kB
> SReclaimable:       6300 kB
> SUnreclaim:        20532 kB
> KernelStack:        3088 kB
> PageTables:          404 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:     8171008 kB
> Committed_AS:      34336 kB
> VmallocTotal:   258998208 kB
> VmallocUsed:           0 kB
> VmallocChunk:          0 kB
> AnonHugePages:         0 kB
> CmaTotal:         204800 kB
> CmaFree:         6602584 kB
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
>


I played with this a bit and can see the same problem. The sanity
check of CmaFree < CmaTotal generally triggers in
__move_zone_freepage_state in unset_migratetype_isolate.
This also seems to be present as far back as v4.0 which was the
first version to have the updated accounting from Joonsoo.
Were there known limitations with the new freepage accounting,
Joonsoo?

Thanks,
Laura

[toc] | [prev] | [next] | [standalone]

#1348809

Hi Laura,

Thanks a lot for taking a look!

On 2016/3/3 9:25, Laura Abbott wrote:
> (cc -mm and Joonsoo Kim)
>
>
[...]
>
> I played with this a bit and can see the same problem. The sanity
> check of CmaFree < CmaTotal generally triggers in
> __move_zone_freepage_state in unset_migratetype_isolate.
> This also seems to be present as far back as v4.0 which was the
> first version to have the updated accounting from Joonsoo.

Would you mind point out the specific commit ID?

Thanks
Hanjun

[toc] | [prev] | [next] | [standalone]

#1348850

2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
> (cc -mm and Joonsoo Kim)
>
>
> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>
>> Hi,
>>
>> I came across a suspicious error for CMA stress test:
>>
>> Before the test, I got:
>> -bash-4.3# cat /proc/meminfo | grep Cma
>> CmaTotal:         204800 kB
>> CmaFree:          195044 kB
>>
>>
>> After running the test:
>> -bash-4.3# cat /proc/meminfo | grep Cma
>> CmaTotal:         204800 kB
>> CmaFree:         6602584 kB
>>
>> So the freed CMA memory is more than total..
>>
>> Also the the MemFree is more than mem total:
>>
>> -bash-4.3# cat /proc/meminfo
>> MemTotal:       16342016 kB
>> MemFree:        22367268 kB
>> MemAvailable:   22370528 kB
>>
>> Here is the kernel module doing the stress test below (if the test case
>> is wrong, correct me), any help would be great appreciated.
>>
>> The test is running on ARM64 platform (hisilicon D02) with 4.4 kernel, I
>> think
>> the 4.5-rc is the same as I didn't notice the updates for it.
>>
>> int malloc_dma(void *data)
>> {
>>      void *vaddr;
>>      struct platform_device * pdev=(struct platform_device*)data;
>>      dma_addr_t dma_handle;
>>      int i;
>>
>>      for(i=0; i<1000; i++) {
>>          vaddr=dma_alloc_coherent(&pdev->dev, malloc_size, &dma_handle,
>> GFP_KERNEL);
>>          if (!vaddr)
>>              pr_err("alloc cma memory failed!\n");
>>
>>          mdelay(1);
>>
>>          if (vaddr)
>>                  dma_free_coherent(&pdev->dev,malloc_size,vaddr,
>> dma_handle);
>>      }
>>      pr_info("alloc free cma memory success return!\n");
>>      return 0;
>> }
>>
>> static int dma_alloc_coherent_init(struct platform_device *pdev)
>> {
>>      int i;
>>
>>      for(i=0; i<100; i++)   {
>>          task[i] = kthread_create(malloc_dma,pdev,"malloc_dma_%d",i);
>>          if(!task[i]) {
>>              printk("kthread_create faile %d\n",i);
>>              continue;
>>          }
>>          wake_up_process(task[i]);
>>      }
>>      return 0;
>> }
>>
>> Thanks
>> Hanjun
>>
>> The whole /proc/meminfo:
>>
>> -bash-4.3# cat /proc/meminfo
>> MemTotal:       16342016 kB
>> MemFree:        22367268 kB
>> MemAvailable:   22370528 kB
>> Buffers:            4292 kB
>> Cached:            36444 kB
>> SwapCached:            0 kB
>> Active:            23564 kB
>> Inactive:          25360 kB
>> Active(anon):       8424 kB
>> Inactive(anon):       64 kB
>> Active(file):      15140 kB
>> Inactive(file):    25296 kB
>> Unevictable:           0 kB
>> Mlocked:               0 kB
>> SwapTotal:             0 kB
>> SwapFree:              0 kB
>> Dirty:                 0 kB
>> Writeback:             0 kB
>> AnonPages:          8196 kB
>> Mapped:            16448 kB
>> Shmem:               296 kB
>> Slab:              26832 kB
>> SReclaimable:       6300 kB
>> SUnreclaim:        20532 kB
>> KernelStack:        3088 kB
>> PageTables:          404 kB
>> NFS_Unstable:          0 kB
>> Bounce:                0 kB
>> WritebackTmp:          0 kB
>> CommitLimit:     8171008 kB
>> Committed_AS:      34336 kB
>> VmallocTotal:   258998208 kB
>> VmallocUsed:           0 kB
>> VmallocChunk:          0 kB
>> AnonHugePages:         0 kB
>> CmaTotal:         204800 kB
>> CmaFree:         6602584 kB
>> HugePages_Total:       0
>> HugePages_Free:        0
>> HugePages_Rsvd:        0
>> HugePages_Surp:        0
>> Hugepagesize:       2048 kB
>>
>
>
> I played with this a bit and can see the same problem. The sanity
> check of CmaFree < CmaTotal generally triggers in
> __move_zone_freepage_state in unset_migratetype_isolate.
> This also seems to be present as far back as v4.0 which was the
> first version to have the updated accounting from Joonsoo.
> Were there known limitations with the new freepage accounting,
> Joonsoo?

I don't know. I also played with this and looks like there is
accounting problem, however, for my case, number of free page is slightly less
than total. I will take a look.

Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
look like your case.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1348852

On 2016/3/3 15:42, Joonsoo Kim wrote:
> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>> (cc -mm and Joonsoo Kim)
>>
>>
>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>> Hi,
>>>
>>> I came across a suspicious error for CMA stress test:
>>>
>>> Before the test, I got:
>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>> CmaTotal:         204800 kB
>>> CmaFree:          195044 kB
>>>
>>>
>>> After running the test:
>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>> CmaTotal:         204800 kB
>>> CmaFree:         6602584 kB
>>>
>>> So the freed CMA memory is more than total..
>>>
>>> Also the the MemFree is more than mem total:
>>>
>>> -bash-4.3# cat /proc/meminfo
>>> MemTotal:       16342016 kB
>>> MemFree:        22367268 kB
>>> MemAvailable:   22370528 kB
>>>
>>> Here is the kernel module doing the stress test below (if the test case
>>> is wrong, correct me), any help would be great appreciated.
>>>
>>> The test is running on ARM64 platform (hisilicon D02) with 4.4 kernel, I
>>> think
>>> the 4.5-rc is the same as I didn't notice the updates for it.
>>>
>>> int malloc_dma(void *data)
>>> {
>>>      void *vaddr;
>>>      struct platform_device * pdev=(struct platform_device*)data;
>>>      dma_addr_t dma_handle;
>>>      int i;
>>>
>>>      for(i=0; i<1000; i++) {
>>>          vaddr=dma_alloc_coherent(&pdev->dev, malloc_size, &dma_handle,
>>> GFP_KERNEL);
>>>          if (!vaddr)
>>>              pr_err("alloc cma memory failed!\n");
>>>
>>>          mdelay(1);
>>>
>>>          if (vaddr)
>>>                  dma_free_coherent(&pdev->dev,malloc_size,vaddr,
>>> dma_handle);
>>>      }
>>>      pr_info("alloc free cma memory success return!\n");
>>>      return 0;
>>> }
>>>
>>> static int dma_alloc_coherent_init(struct platform_device *pdev)
>>> {
>>>      int i;
>>>
>>>      for(i=0; i<100; i++)   {
>>>          task[i] = kthread_create(malloc_dma,pdev,"malloc_dma_%d",i);
>>>          if(!task[i]) {
>>>              printk("kthread_create faile %d\n",i);
>>>              continue;
>>>          }
>>>          wake_up_process(task[i]);
>>>      }
>>>      return 0;
>>> }
>>>
>>> Thanks
>>> Hanjun
>>>
>>> The whole /proc/meminfo:
>>>
>>> -bash-4.3# cat /proc/meminfo
>>> MemTotal:       16342016 kB
>>> MemFree:        22367268 kB
>>> MemAvailable:   22370528 kB
>>> Buffers:            4292 kB
>>> Cached:            36444 kB
>>> SwapCached:            0 kB
>>> Active:            23564 kB
>>> Inactive:          25360 kB
>>> Active(anon):       8424 kB
>>> Inactive(anon):       64 kB
>>> Active(file):      15140 kB
>>> Inactive(file):    25296 kB
>>> Unevictable:           0 kB
>>> Mlocked:               0 kB
>>> SwapTotal:             0 kB
>>> SwapFree:              0 kB
>>> Dirty:                 0 kB
>>> Writeback:             0 kB
>>> AnonPages:          8196 kB
>>> Mapped:            16448 kB
>>> Shmem:               296 kB
>>> Slab:              26832 kB
>>> SReclaimable:       6300 kB
>>> SUnreclaim:        20532 kB
>>> KernelStack:        3088 kB
>>> PageTables:          404 kB
>>> NFS_Unstable:          0 kB
>>> Bounce:                0 kB
>>> WritebackTmp:          0 kB
>>> CommitLimit:     8171008 kB
>>> Committed_AS:      34336 kB
>>> VmallocTotal:   258998208 kB
>>> VmallocUsed:           0 kB
>>> VmallocChunk:          0 kB
>>> AnonHugePages:         0 kB
>>> CmaTotal:         204800 kB
>>> CmaFree:         6602584 kB
>>> HugePages_Total:       0
>>> HugePages_Free:        0
>>> HugePages_Rsvd:        0
>>> HugePages_Surp:        0
>>> Hugepagesize:       2048 kB
>>>
>>
>> I played with this a bit and can see the same problem. The sanity
>> check of CmaFree < CmaTotal generally triggers in
>> __move_zone_freepage_state in unset_migratetype_isolate.
>> This also seems to be present as far back as v4.0 which was the
>> first version to have the updated accounting from Joonsoo.
>> Were there known limitations with the new freepage accounting,
>> Joonsoo?
> I don't know. I also played with this and looks like there is
> accounting problem, however, for my case, number of free page is slightly less
> than total. I will take a look.
>
> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
> look like your case.

 The malloc_size is 1M, and with 200M total (passed via boot commandline cma=200M),
any more information is needed, please let me know.

Thanks for the help!
Hanjun

[toc] | [prev] | [next] | [standalone]

#1349127

On 2016/3/3 15:42, Joonsoo Kim wrote:
> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>> (cc -mm and Joonsoo Kim)
>>
>>
>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>> Hi,
>>>
>>> I came across a suspicious error for CMA stress test:
>>>
>>> Before the test, I got:
>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>> CmaTotal:         204800 kB
>>> CmaFree:          195044 kB
>>>
>>>
>>> After running the test:
>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>> CmaTotal:         204800 kB
>>> CmaFree:         6602584 kB
>>>
>>> So the freed CMA memory is more than total..
>>>
>>> Also the the MemFree is more than mem total:
>>>
>>> -bash-4.3# cat /proc/meminfo
>>> MemTotal:       16342016 kB
>>> MemFree:        22367268 kB
>>> MemAvailable:   22370528 kB
[...]
>>
>> I played with this a bit and can see the same problem. The sanity
>> check of CmaFree < CmaTotal generally triggers in
>> __move_zone_freepage_state in unset_migratetype_isolate.
>> This also seems to be present as far back as v4.0 which was the
>> first version to have the updated accounting from Joonsoo.
>> Were there known limitations with the new freepage accounting,
>> Joonsoo?
> I don't know. I also played with this and looks like there is
> accounting problem, however, for my case, number of free page is slightly less
> than total. I will take a look.
>
> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
> look like your case.

I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
did some other test:

 - run with single thread with 100000 times, everything is fine.

 - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
   the same test with 100 multi-thread, then I got:

-bash-4.3# cat /proc/meminfo | grep Cma
CmaTotal: 204800 kB
CmaFree: 225112 kB

It only increased about 30M for free, not 6G+ in previous test, although
the problem is not solved, the problem is less serious, is it a synchronization
problem?

Thanks
Hanjun

[1]:
index ea506eb..4447494 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -379,6 +379,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align)
        if (!count)
                return NULL;
 
+ mutex_lock(&cma_mutex);
        mask = cma_bitmap_aligned_mask(cma, align);
        offset = cma_bitmap_aligned_offset(cma, align);
        bitmap_maxno = cma_bitmap_maxno(cma);
@@ -402,17 +403,16 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align)
                mutex_unlock(&cma->lock);
 
                pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit);
-           mutex_lock(&cma_mutex);
                ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
-           mutex_unlock(&cma_mutex);
                if (ret == 0) {
                        page = pfn_to_page(pfn);
                        break;
                }
 
                cma_clear_bitmap(cma, pfn, count);
-           if (ret != -EBUSY)
+         if (ret != -EBUSY) {
                        break;
+         }
 
                pr_debug("%s(): memory range at %p is busy, retrying\n",
                         __func__, pfn_to_page(pfn));
@@ -420,6 +420,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align)
                start = bitmap_no + mask + 1;
        }
 
+ mutex_unlock(&cma_mutex);
        trace_cma_alloc(pfn, page, count, align);
 
        pr_debug("%s(): returned %p\n", __func__, page);
@@ -445,15 +446,19 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count)
 
        pr_debug("%s(page %p)\n", __func__, (void *)pages);
 
+ mutex_lock(&cma_mutex);
        pfn = page_to_pfn(pages);
 
-   if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count)
+ if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) {
+         mutex_unlock(&cma_mutex);
                return false;
+ }
 
        VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
 
        free_contig_range(pfn, count);
        cma_clear_bitmap(cma, pfn, count);
+ mutex_unlock(&cma_mutex);
        trace_cma_release(pfn, pages, count);
 
        return true;

[toc] | [prev] | [next] | [standalone]

#1349489

On 03/03/2016 04:49 AM, Hanjun Guo wrote:
> On 2016/3/3 15:42, Joonsoo Kim wrote:
>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>>> (cc -mm and Joonsoo Kim)
>>>
>>>
>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>>> Hi,
>>>>
>>>> I came across a suspicious error for CMA stress test:
>>>>
>>>> Before the test, I got:
>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>> CmaTotal:         204800 kB
>>>> CmaFree:          195044 kB
>>>>
>>>>
>>>> After running the test:
>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>> CmaTotal:         204800 kB
>>>> CmaFree:         6602584 kB
>>>>
>>>> So the freed CMA memory is more than total..
>>>>
>>>> Also the the MemFree is more than mem total:
>>>>
>>>> -bash-4.3# cat /proc/meminfo
>>>> MemTotal:       16342016 kB
>>>> MemFree:        22367268 kB
>>>> MemAvailable:   22370528 kB
> [...]
>>>
>>> I played with this a bit and can see the same problem. The sanity
>>> check of CmaFree < CmaTotal generally triggers in
>>> __move_zone_freepage_state in unset_migratetype_isolate.
>>> This also seems to be present as far back as v4.0 which was the
>>> first version to have the updated accounting from Joonsoo.
>>> Were there known limitations with the new freepage accounting,
>>> Joonsoo?
>> I don't know. I also played with this and looks like there is
>> accounting problem, however, for my case, number of free page is slightly less
>> than total. I will take a look.
>>
>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
>> look like your case.
>
> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
> did some other test:
>
>   - run with single thread with 100000 times, everything is fine.
>
>   - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>     the same test with 100 multi-thread, then I got:
>
> -bash-4.3# cat /proc/meminfo | grep Cma
> CmaTotal: 204800 kB
> CmaFree: 225112 kB
>
> It only increased about 30M for free, not 6G+ in previous test, although
> the problem is not solved, the problem is less serious, is it a synchronization
> problem?
>

'only' 30M is still an issue although I think you are right about something related
to synchronization. When I put the cma_mutex around free_contig_range I don't see
the issue. I wonder if free of the pages is racing with the undo_isolate_page_range
on overlapping ranges caused by outer_start?

Thanks,
Laura


> Thanks
> Hanjun
>
> [1]:
> index ea506eb..4447494 100644
> --- a/mm/cma.c
> +++ b/mm/cma.c
> @@ -379,6 +379,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align)
>          if (!count)
>                  return NULL;
>
> + mutex_lock(&cma_mutex);
>          mask = cma_bitmap_aligned_mask(cma, align);
>          offset = cma_bitmap_aligned_offset(cma, align);
>          bitmap_maxno = cma_bitmap_maxno(cma);
> @@ -402,17 +403,16 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align)
>                  mutex_unlock(&cma->lock);
>
>                  pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit);
> -           mutex_lock(&cma_mutex);
>                  ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
> -           mutex_unlock(&cma_mutex);
>                  if (ret == 0) {
>                          page = pfn_to_page(pfn);
>                          break;
>                  }
>
>                  cma_clear_bitmap(cma, pfn, count);
> -           if (ret != -EBUSY)
> +         if (ret != -EBUSY) {
>                          break;
> +         }
>
>                  pr_debug("%s(): memory range at %p is busy, retrying\n",
>                           __func__, pfn_to_page(pfn));
> @@ -420,6 +420,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align)
>                  start = bitmap_no + mask + 1;
>          }
>
> + mutex_unlock(&cma_mutex);
>          trace_cma_alloc(pfn, page, count, align);
>
>          pr_debug("%s(): returned %p\n", __func__, page);
> @@ -445,15 +446,19 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count)
>
>          pr_debug("%s(page %p)\n", __func__, (void *)pages);
>
> + mutex_lock(&cma_mutex);
>          pfn = page_to_pfn(pages);
>
> -   if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count)
> + if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) {
> +         mutex_unlock(&cma_mutex);
>                  return false;
> + }
>
>          VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
>
>          free_contig_range(pfn, count);
>          cma_clear_bitmap(cma, pfn, count);
> + mutex_unlock(&cma_mutex);
>          trace_cma_release(pfn, pages, count);
>
>          return true;
>

[toc] | [prev] | [next] | [standalone]

#1349782

On Thu, Mar 03, 2016 at 10:52:17AM -0800, Laura Abbott wrote:
> On 03/03/2016 04:49 AM, Hanjun Guo wrote:
> >On 2016/3/3 15:42, Joonsoo Kim wrote:
> >>2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
> >>>(cc -mm and Joonsoo Kim)
> >>>
> >>>
> >>>On 03/02/2016 05:52 AM, Hanjun Guo wrote:
> >>>>Hi,
> >>>>
> >>>>I came across a suspicious error for CMA stress test:
> >>>>
> >>>>Before the test, I got:
> >>>>-bash-4.3# cat /proc/meminfo | grep Cma
> >>>>CmaTotal:         204800 kB
> >>>>CmaFree:          195044 kB
> >>>>
> >>>>
> >>>>After running the test:
> >>>>-bash-4.3# cat /proc/meminfo | grep Cma
> >>>>CmaTotal:         204800 kB
> >>>>CmaFree:         6602584 kB
> >>>>
> >>>>So the freed CMA memory is more than total..
> >>>>
> >>>>Also the the MemFree is more than mem total:
> >>>>
> >>>>-bash-4.3# cat /proc/meminfo
> >>>>MemTotal:       16342016 kB
> >>>>MemFree:        22367268 kB
> >>>>MemAvailable:   22370528 kB
> >[...]
> >>>
> >>>I played with this a bit and can see the same problem. The sanity
> >>>check of CmaFree < CmaTotal generally triggers in
> >>>__move_zone_freepage_state in unset_migratetype_isolate.
> >>>This also seems to be present as far back as v4.0 which was the
> >>>first version to have the updated accounting from Joonsoo.
> >>>Were there known limitations with the new freepage accounting,
> >>>Joonsoo?
> >>I don't know. I also played with this and looks like there is
> >>accounting problem, however, for my case, number of free page is slightly less
> >>than total. I will take a look.
> >>
> >>Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
> >>look like your case.
> >
> >I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
> >did some other test:
> >
> >  - run with single thread with 100000 times, everything is fine.
> >
> >  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
> >    the same test with 100 multi-thread, then I got:
> >
> >-bash-4.3# cat /proc/meminfo | grep Cma
> >CmaTotal: 204800 kB
> >CmaFree: 225112 kB
> >
> >It only increased about 30M for free, not 6G+ in previous test, although
> >the problem is not solved, the problem is less serious, is it a synchronization
> >problem?
> >
> 
> 'only' 30M is still an issue although I think you are right about something related
> to synchronization. When I put the cma_mutex around free_contig_range I don't see

Hmm... I can see the issue even if putting the cma_mutex around
free_contig_range().

In other reply, I attached the code to temporary close the race.

> the issue. I wonder if free of the pages is racing with the undo_isolate_page_range
> on overlapping ranges caused by outer_start?

I don't know yet.
Anyway, it looks like that the problem that I want to fix by commit '3c60509'
still remains.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1349873

On 2016/3/4 10:09, Joonsoo Kim wrote:
> On Thu, Mar 03, 2016 at 10:52:17AM -0800, Laura Abbott wrote:
>> On 03/03/2016 04:49 AM, Hanjun Guo wrote:
>>> On 2016/3/3 15:42, Joonsoo Kim wrote:
>>>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>>>>> (cc -mm and Joonsoo Kim)
>>>>>
>>>>>
>>>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I came across a suspicious error for CMA stress test:
>>>>>>
>>>>>> Before the test, I got:
>>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>>> CmaTotal:         204800 kB
>>>>>> CmaFree:          195044 kB
>>>>>>
>>>>>>
>>>>>> After running the test:
>>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>>> CmaTotal:         204800 kB
>>>>>> CmaFree:         6602584 kB
>>>>>>
>>>>>> So the freed CMA memory is more than total..
>>>>>>
>>>>>> Also the the MemFree is more than mem total:
>>>>>>
>>>>>> -bash-4.3# cat /proc/meminfo
>>>>>> MemTotal:       16342016 kB
>>>>>> MemFree:        22367268 kB
>>>>>> MemAvailable:   22370528 kB
>>> [...]
>>>>> I played with this a bit and can see the same problem. The sanity
>>>>> check of CmaFree < CmaTotal generally triggers in
>>>>> __move_zone_freepage_state in unset_migratetype_isolate.
>>>>> This also seems to be present as far back as v4.0 which was the
>>>>> first version to have the updated accounting from Joonsoo.
>>>>> Were there known limitations with the new freepage accounting,
>>>>> Joonsoo?
>>>> I don't know. I also played with this and looks like there is
>>>> accounting problem, however, for my case, number of free page is slightly less
>>>> than total. I will take a look.
>>>>
>>>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
>>>> look like your case.
>>> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
>>> did some other test:
>>>
>>>  - run with single thread with 100000 times, everything is fine.
>>>
>>>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>>>    the same test with 100 multi-thread, then I got:
>>>
>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>> CmaTotal: 204800 kB
>>> CmaFree: 225112 kB
>>>
>>> It only increased about 30M for free, not 6G+ in previous test, although
>>> the problem is not solved, the problem is less serious, is it a synchronization
>>> problem?
>>>
>> 'only' 30M is still an issue although I think you are right about something related
>> to synchronization. When I put the cma_mutex around free_contig_range I don't see
> Hmm... I can see the issue even if putting the cma_mutex around
> free_contig_range().

Yes, I can confirm that too, it can reduce the number of erronous freed memory, but
the problem is still there.

Thanks
Hanjun

[toc] | [prev] | [next] | [standalone]

#1349781

On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
> On 2016/3/3 15:42, Joonsoo Kim wrote:
> > 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
> >> (cc -mm and Joonsoo Kim)
> >>
> >>
> >> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
> >>> Hi,
> >>>
> >>> I came across a suspicious error for CMA stress test:
> >>>
> >>> Before the test, I got:
> >>> -bash-4.3# cat /proc/meminfo | grep Cma
> >>> CmaTotal:         204800 kB
> >>> CmaFree:          195044 kB
> >>>
> >>>
> >>> After running the test:
> >>> -bash-4.3# cat /proc/meminfo | grep Cma
> >>> CmaTotal:         204800 kB
> >>> CmaFree:         6602584 kB
> >>>
> >>> So the freed CMA memory is more than total..
> >>>
> >>> Also the the MemFree is more than mem total:
> >>>
> >>> -bash-4.3# cat /proc/meminfo
> >>> MemTotal:       16342016 kB
> >>> MemFree:        22367268 kB
> >>> MemAvailable:   22370528 kB
> [...]
> >>
> >> I played with this a bit and can see the same problem. The sanity
> >> check of CmaFree < CmaTotal generally triggers in
> >> __move_zone_freepage_state in unset_migratetype_isolate.
> >> This also seems to be present as far back as v4.0 which was the
> >> first version to have the updated accounting from Joonsoo.
> >> Were there known limitations with the new freepage accounting,
> >> Joonsoo?
> > I don't know. I also played with this and looks like there is
> > accounting problem, however, for my case, number of free page is slightly less
> > than total. I will take a look.
> >
> > Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
> > look like your case.
> 
> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
> did some other test:

Thanks! Now, I can re-generate erronous situation you mentioned.

> 
>  - run with single thread with 100000 times, everything is fine.
> 
>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>    the same test with 100 multi-thread, then I got:

[1] would not be sufficient to close this race.

Try following things [A]. And, for more accurate test, I changed code a bit more
to prevent kernel page allocation from cma area [B]. This will prevent kernel
page allocation from cma area completely so we can focus cma_alloc/release race.

Although, this is not correct fix, it could help that we can guess
where the problem is.

Thanks.

[A]
diff --git a/mm/cma.c b/mm/cma.c
index c003274..43ed02d 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -496,7 +496,9 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count)
 
        VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
 
+       mutex_lock(&cma_mutex);
        free_contig_range(pfn, count);
+       mutex_unlock(&cma_mutex);
        cma_clear_bitmap(cma, pfn, count);
        trace_cma_release(pfn, pages, count);
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c6c38ed..1ce8a59 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2192,7 +2192,8 @@ void free_hot_cold_page(struct page *page, bool cold)
         * excessively into the page allocator
         */
        if (migratetype >= MIGRATE_PCPTYPES) {
-               if (unlikely(is_migrate_isolate(migratetype))) {
+               if (is_migrate_cma(migratetype) ||
+                       unlikely(is_migrate_isolate(migratetype))) {
                        free_one_page(zone, page, pfn, 0, migratetype);
                        goto out;
                }


[B]
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f2dccf9..c6c38ed 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1493,6 +1493,7 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
                                                                int alloc_flags)
 {
        int i;
+       bool cma = false;
 
        for (i = 0; i < (1 << order); i++) {
                struct page *p = page + i;
@@ -1500,6 +1501,9 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
                        return 1;
        }
 
+       if (is_migrate_cma(get_pcppage_migratetype(page)))
+               cma = true;
+
        set_page_private(page, 0);
        set_page_refcounted(page);
 
@@ -1528,6 +1532,12 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
        else
                clear_page_pfmemalloc(page);
 
+       if (cma) {
+               page_ref_dec(page);
+               __free_pages_ok(page, order);
+               return 1;
+       }
+
        return 0;
 }
 
@@ -1582,7 +1592,7 @@ static int fallbacks[MIGRATE_TYPES][4] = {
 static struct page *__rmqueue_cma_fallback(struct zone *zone,
                                        unsigned int order)
 {
-       return __rmqueue_smallest(zone, order, MIGRATE_CMA);
+       return NULL;
 }
 #else
 static inline struct page *__rmqueue_cma_fallback(struct zone *zone,

[toc] | [prev] | [next] | [standalone]

#1349831

On Fri, Mar 04, 2016 at 11:02:33AM +0900, Joonsoo Kim wrote:
> On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
> > On 2016/3/3 15:42, Joonsoo Kim wrote:
> > > 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
> > >> (cc -mm and Joonsoo Kim)
> > >>
> > >>
> > >> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
> > >>> Hi,
> > >>>
> > >>> I came across a suspicious error for CMA stress test:
> > >>>
> > >>> Before the test, I got:
> > >>> -bash-4.3# cat /proc/meminfo | grep Cma
> > >>> CmaTotal:         204800 kB
> > >>> CmaFree:          195044 kB
> > >>>
> > >>>
> > >>> After running the test:
> > >>> -bash-4.3# cat /proc/meminfo | grep Cma
> > >>> CmaTotal:         204800 kB
> > >>> CmaFree:         6602584 kB
> > >>>
> > >>> So the freed CMA memory is more than total..
> > >>>
> > >>> Also the the MemFree is more than mem total:
> > >>>
> > >>> -bash-4.3# cat /proc/meminfo
> > >>> MemTotal:       16342016 kB
> > >>> MemFree:        22367268 kB
> > >>> MemAvailable:   22370528 kB
> > [...]
> > >>
> > >> I played with this a bit and can see the same problem. The sanity
> > >> check of CmaFree < CmaTotal generally triggers in
> > >> __move_zone_freepage_state in unset_migratetype_isolate.
> > >> This also seems to be present as far back as v4.0 which was the
> > >> first version to have the updated accounting from Joonsoo.
> > >> Were there known limitations with the new freepage accounting,
> > >> Joonsoo?
> > > I don't know. I also played with this and looks like there is
> > > accounting problem, however, for my case, number of free page is slightly less
> > > than total. I will take a look.
> > >
> > > Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
> > > look like your case.
> > 
> > I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
> > did some other test:
> 
> Thanks! Now, I can re-generate erronous situation you mentioned.
> 
> > 
> >  - run with single thread with 100000 times, everything is fine.
> > 
> >  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
> >    the same test with 100 multi-thread, then I got:
> 
> [1] would not be sufficient to close this race.
> 
> Try following things [A]. And, for more accurate test, I changed code a bit more
> to prevent kernel page allocation from cma area [B]. This will prevent kernel
> page allocation from cma area completely so we can focus cma_alloc/release race.
> 
> Although, this is not correct fix, it could help that we can guess
> where the problem is.

More correct fix is something like below.
Please test it.

It checks problematic buddy merging and prevent it.
I will try to find another way that is less intrusive for freepath performance.

Thanks.

---------------->8-----------------------
From 855cb11368487a0f02a5ad5b3d9de375dfbb061c Mon Sep 17 00:00:00 2001
From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Date: Fri, 4 Mar 2016 13:28:17 +0900
Subject: [PATCH] mm/cma: fix race

Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
 mm/page_alloc.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index c6c38ed..a01c3b5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -620,8 +620,8 @@ static inline void rmv_page_order(struct page *page)
  *
  * For recording page's order, we use page_private(page).
  */
-static inline int page_is_buddy(struct page *page, struct page *buddy,
-                                                       unsigned int order)
+static inline int page_is_buddy(struct zone *zone, struct page *page,
+                               struct page *buddy, unsigned int order)
 {
        if (!pfn_valid_within(page_to_pfn(buddy)))
                return 0;
@@ -644,6 +644,12 @@ static inline int page_is_buddy(struct page *page, struct page *buddy,
                if (page_zone_id(page) != page_zone_id(buddy))
                        return 0;
 
+               if (IS_ENABLED(CONFIG_CMA) &&
+                       has_isolate_pageblock(zone) &&
+                       order >= pageblock_order &&
+                       is_migrate_isolate(get_pageblock_migratetype(buddy)))
+                       return 0;
+
                VM_BUG_ON_PAGE(page_count(buddy) != 0, buddy);
 
                return 1;
@@ -711,7 +717,7 @@ static inline void __free_one_page(struct page *page,
        while (order < max_order - 1) {
                buddy_idx = __find_buddy_index(page_idx, order);
                buddy = page + (buddy_idx - page_idx);
-               if (!page_is_buddy(page, buddy, order))
+               if (!page_is_buddy(zone, page, buddy, order))
                        break;
                /*
                 * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page,
@@ -745,7 +751,7 @@ static inline void __free_one_page(struct page *page,
                higher_page = page + (combined_idx - page_idx);
                buddy_idx = __find_buddy_index(combined_idx, order + 1);
                higher_buddy = higher_page + (buddy_idx - combined_idx);
-               if (page_is_buddy(higher_page, higher_buddy, order + 1)) {
+               if (page_is_buddy(zone, higher_page, higher_buddy, order + 1)) {
                        list_add_tail(&page->lru,
                                &zone->free_area[order].free_list[migratetype]);
                        goto out;
-- 
1.9.1

[toc] | [prev] | [next] | [standalone]

#1349872

On 2016/3/4 12:32, Joonsoo Kim wrote:
> On Fri, Mar 04, 2016 at 11:02:33AM +0900, Joonsoo Kim wrote:
>> On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
>>> On 2016/3/3 15:42, Joonsoo Kim wrote:
>>>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>>>>> (cc -mm and Joonsoo Kim)
>>>>>
>>>>>
>>>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I came across a suspicious error for CMA stress test:
>>>>>>
>>>>>> Before the test, I got:
>>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>>> CmaTotal:         204800 kB
>>>>>> CmaFree:          195044 kB
>>>>>>
>>>>>>
>>>>>> After running the test:
>>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>>> CmaTotal:         204800 kB
>>>>>> CmaFree:         6602584 kB
>>>>>>
>>>>>> So the freed CMA memory is more than total..
>>>>>>
>>>>>> Also the the MemFree is more than mem total:
>>>>>>
>>>>>> -bash-4.3# cat /proc/meminfo
>>>>>> MemTotal:       16342016 kB
>>>>>> MemFree:        22367268 kB
>>>>>> MemAvailable:   22370528 kB
>>> [...]
>>>>> I played with this a bit and can see the same problem. The sanity
>>>>> check of CmaFree < CmaTotal generally triggers in
>>>>> __move_zone_freepage_state in unset_migratetype_isolate.
>>>>> This also seems to be present as far back as v4.0 which was the
>>>>> first version to have the updated accounting from Joonsoo.
>>>>> Were there known limitations with the new freepage accounting,
>>>>> Joonsoo?
>>>> I don't know. I also played with this and looks like there is
>>>> accounting problem, however, for my case, number of free page is slightly less
>>>> than total. I will take a look.
>>>>
>>>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
>>>> look like your case.
>>> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
>>> did some other test:
>> Thanks! Now, I can re-generate erronous situation you mentioned.
>>
>>>  - run with single thread with 100000 times, everything is fine.
>>>
>>>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>>>    the same test with 100 multi-thread, then I got:
>> [1] would not be sufficient to close this race.
>>
>> Try following things [A]. And, for more accurate test, I changed code a bit more
>> to prevent kernel page allocation from cma area [B]. This will prevent kernel
>> page allocation from cma area completely so we can focus cma_alloc/release race.
>>
>> Although, this is not correct fix, it could help that we can guess
>> where the problem is.
> More correct fix is something like below.
> Please test it.

Hmm, this is not working:

-bash-4.3# cat /proc/meminfo  |grep Cma                                                                                            
CmaTotal:         204800 kB                                                                                                        
CmaFree:        19388216 kB

-bash-4.3# cat /proc/meminfo                                                                                                       
MemTotal:       16342016 kB                                                                                                        
MemFree:        35146212 kB                                                                                                        
MemAvailable:   35158008 kB                                                                                                        
Buffers:            4236 kB                                                                                                        
Cached:            45032 kB                                                                                                        
SwapCached:            0 kB                                                                                                        
Active:            19276 kB                                                                                                        
Inactive:          36492 kB                                                                                                        
Active(anon):       6724 kB                                                                                                        
Inactive(anon):       52 kB                                                                                                        
Active(file):      12552 kB                                                                                                        
Inactive(file):    36440 kB                                                                                                        
Unevictable:           0 kB                                                                                                        
Mlocked:               0 kB                                                                                                        
SwapTotal:             0 kB                                                                                                        
SwapFree:              0 kB                                                                                                        
Dirty:                 0 kB                                                                                                        
Writeback:             0 kB                                                                                                        
AnonPages:          6524 kB                                                                                                        
Mapped:            24724 kB                                                                                                        
Shmem:               264 kB                                                                                                        
Slab:              26948 kB                                                                                                        
SReclaimable:       6260 kB                                                                                                        
SUnreclaim:        20688 kB                                                                                                        
KernelStack:        3296 kB                                                                                                        
PageTables:          400 kB                                                                                                        
NFS_Unstable:          0 kB                                                                                                        
Bounce:                0 kB                                                                                                        
WritebackTmp:          0 kB                                                                                                        
CommitLimit:     8171008 kB                                                                                                        
Committed_AS:      32764 kB                                                                                                        
VmallocTotal:   258998208 kB                                                                                                       
VmallocUsed:           0 kB                                                                                                        
VmallocChunk:          0 kB                                                                                                        
AnonHugePages:         0 kB                                                                                                        
CmaTotal:         204800 kB                                                                                                        
CmaFree:        19388216 kB                                                                                                        
HugePages_Total:       0                                                                                                           
HugePages_Free:        0                                                                                                           
HugePages_Rsvd:        0                                                                                                           
HugePages_Surp:        0                                                                                                           
Hugepagesize:       2048 kB

Thanks
Hanjun

>
> It checks problematic buddy merging and prevent it.
> I will try to find another way that is less intrusive for freepath performance.
>
> Thanks.
>
> ---------------->8-----------------------
> >From 855cb11368487a0f02a5ad5b3d9de375dfbb061c Mon Sep 17 00:00:00 2001
> From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> Date: Fri, 4 Mar 2016 13:28:17 +0900
> Subject: [PATCH] mm/cma: fix race
>
> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> ---
>  mm/page_alloc.c | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index c6c38ed..a01c3b5 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -620,8 +620,8 @@ static inline void rmv_page_order(struct page *page)
>   *
>   * For recording page's order, we use page_private(page).
>   */
> -static inline int page_is_buddy(struct page *page, struct page *buddy,
> -                                                       unsigned int order)
> +static inline int page_is_buddy(struct zone *zone, struct page *page,
> +                               struct page *buddy, unsigned int order)
>  {
>         if (!pfn_valid_within(page_to_pfn(buddy)))
>                 return 0;
> @@ -644,6 +644,12 @@ static inline int page_is_buddy(struct page *page, struct page *buddy,
>                 if (page_zone_id(page) != page_zone_id(buddy))
>                         return 0;
>  
> +               if (IS_ENABLED(CONFIG_CMA) &&
> +                       has_isolate_pageblock(zone) &&
> +                       order >= pageblock_order &&
> +                       is_migrate_isolate(get_pageblock_migratetype(buddy)))
> +                       return 0;
> +
>                 VM_BUG_ON_PAGE(page_count(buddy) != 0, buddy);
>  
>                 return 1;
> @@ -711,7 +717,7 @@ static inline void __free_one_page(struct page *page,
>         while (order < max_order - 1) {
>                 buddy_idx = __find_buddy_index(page_idx, order);
>                 buddy = page + (buddy_idx - page_idx);
> -               if (!page_is_buddy(page, buddy, order))
> +               if (!page_is_buddy(zone, page, buddy, order))
>                         break;
>                 /*
>                  * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page,
> @@ -745,7 +751,7 @@ static inline void __free_one_page(struct page *page,
>                 higher_page = page + (combined_idx - page_idx);
>                 buddy_idx = __find_buddy_index(combined_idx, order + 1);
>                 higher_buddy = higher_page + (buddy_idx - combined_idx);
> -               if (page_is_buddy(higher_page, higher_buddy, order + 1)) {
> +               if (page_is_buddy(zone, higher_page, higher_buddy, order + 1)) {
>                         list_add_tail(&page->lru,
>                                 &zone->free_area[order].free_list[migratetype]);
>                         goto out;

[toc] | [prev] | [next] | [standalone]

#1349881

On Fri, Mar 04, 2016 at 02:05:09PM +0800, Hanjun Guo wrote:
> On 2016/3/4 12:32, Joonsoo Kim wrote:
> > On Fri, Mar 04, 2016 at 11:02:33AM +0900, Joonsoo Kim wrote:
> >> On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
> >>> On 2016/3/3 15:42, Joonsoo Kim wrote:
> >>>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
> >>>>> (cc -mm and Joonsoo Kim)
> >>>>>
> >>>>>
> >>>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I came across a suspicious error for CMA stress test:
> >>>>>>
> >>>>>> Before the test, I got:
> >>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
> >>>>>> CmaTotal:         204800 kB
> >>>>>> CmaFree:          195044 kB
> >>>>>>
> >>>>>>
> >>>>>> After running the test:
> >>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
> >>>>>> CmaTotal:         204800 kB
> >>>>>> CmaFree:         6602584 kB
> >>>>>>
> >>>>>> So the freed CMA memory is more than total..
> >>>>>>
> >>>>>> Also the the MemFree is more than mem total:
> >>>>>>
> >>>>>> -bash-4.3# cat /proc/meminfo
> >>>>>> MemTotal:       16342016 kB
> >>>>>> MemFree:        22367268 kB
> >>>>>> MemAvailable:   22370528 kB
> >>> [...]
> >>>>> I played with this a bit and can see the same problem. The sanity
> >>>>> check of CmaFree < CmaTotal generally triggers in
> >>>>> __move_zone_freepage_state in unset_migratetype_isolate.
> >>>>> This also seems to be present as far back as v4.0 which was the
> >>>>> first version to have the updated accounting from Joonsoo.
> >>>>> Were there known limitations with the new freepage accounting,
> >>>>> Joonsoo?
> >>>> I don't know. I also played with this and looks like there is
> >>>> accounting problem, however, for my case, number of free page is slightly less
> >>>> than total. I will take a look.
> >>>>
> >>>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
> >>>> look like your case.
> >>> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
> >>> did some other test:
> >> Thanks! Now, I can re-generate erronous situation you mentioned.
> >>
> >>>  - run with single thread with 100000 times, everything is fine.
> >>>
> >>>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
> >>>    the same test with 100 multi-thread, then I got:
> >> [1] would not be sufficient to close this race.
> >>
> >> Try following things [A]. And, for more accurate test, I changed code a bit more
> >> to prevent kernel page allocation from cma area [B]. This will prevent kernel
> >> page allocation from cma area completely so we can focus cma_alloc/release race.
> >>
> >> Although, this is not correct fix, it could help that we can guess
> >> where the problem is.
> > More correct fix is something like below.
> > Please test it.
> 
> Hmm, this is not working:

Sad to hear that.

Could you tell me your system's MAX_ORDER and pageblock_order?

Thanks.

[toc] | [prev] | [next] | [standalone]

#1349918

On 2016/3/4 14:38, Joonsoo Kim wrote:
> On Fri, Mar 04, 2016 at 02:05:09PM +0800, Hanjun Guo wrote:
>> On 2016/3/4 12:32, Joonsoo Kim wrote:
>>> On Fri, Mar 04, 2016 at 11:02:33AM +0900, Joonsoo Kim wrote:
>>>> On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
>>>>> On 2016/3/3 15:42, Joonsoo Kim wrote:
>>>>>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>>>>>>> (cc -mm and Joonsoo Kim)
>>>>>>>
>>>>>>>
>>>>>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I came across a suspicious error for CMA stress test:
>>>>>>>>
>>>>>>>> Before the test, I got:
>>>>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>>>>> CmaTotal:         204800 kB
>>>>>>>> CmaFree:          195044 kB
>>>>>>>>
>>>>>>>>
>>>>>>>> After running the test:
>>>>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>>>>> CmaTotal:         204800 kB
>>>>>>>> CmaFree:         6602584 kB
>>>>>>>>
>>>>>>>> So the freed CMA memory is more than total..
>>>>>>>>
>>>>>>>> Also the the MemFree is more than mem total:
>>>>>>>>
>>>>>>>> -bash-4.3# cat /proc/meminfo
>>>>>>>> MemTotal:       16342016 kB
>>>>>>>> MemFree:        22367268 kB
>>>>>>>> MemAvailable:   22370528 kB
>>>>> [...]
>>>>>>> I played with this a bit and can see the same problem. The sanity
>>>>>>> check of CmaFree < CmaTotal generally triggers in
>>>>>>> __move_zone_freepage_state in unset_migratetype_isolate.
>>>>>>> This also seems to be present as far back as v4.0 which was the
>>>>>>> first version to have the updated accounting from Joonsoo.
>>>>>>> Were there known limitations with the new freepage accounting,
>>>>>>> Joonsoo?
>>>>>> I don't know. I also played with this and looks like there is
>>>>>> accounting problem, however, for my case, number of free page is slightly less
>>>>>> than total. I will take a look.
>>>>>>
>>>>>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
>>>>>> look like your case.
>>>>> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
>>>>> did some other test:
>>>> Thanks! Now, I can re-generate erronous situation you mentioned.
>>>>
>>>>>  - run with single thread with 100000 times, everything is fine.
>>>>>
>>>>>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>>>>>    the same test with 100 multi-thread, then I got:
>>>> [1] would not be sufficient to close this race.
>>>>
>>>> Try following things [A]. And, for more accurate test, I changed code a bit more
>>>> to prevent kernel page allocation from cma area [B]. This will prevent kernel
>>>> page allocation from cma area completely so we can focus cma_alloc/release race.
>>>>
>>>> Although, this is not correct fix, it could help that we can guess
>>>> where the problem is.
>>> More correct fix is something like below.
>>> Please test it.
>> Hmm, this is not working:
> Sad to hear that.
>
> Could you tell me your system's MAX_ORDER and pageblock_order?
>

MAX_ORDER is 11, pageblock_order is 9, thanks for your help!

Hanjun

[toc] | [prev] | [next] | [standalone]

#1349853

Hi Joonsoo,

On 2016/3/4 10:02, Joonsoo Kim wrote:
> On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
>> On 2016/3/3 15:42, Joonsoo Kim wrote:
>>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>>>> (cc -mm and Joonsoo Kim)
>>>>
>>>>
>>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>>>> Hi,
>>>>>
>>>>> I came across a suspicious error for CMA stress test:
>>>>>
>>>>> Before the test, I got:
>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>> CmaTotal:         204800 kB
>>>>> CmaFree:          195044 kB
>>>>>
>>>>>
>>>>> After running the test:
>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>> CmaTotal:         204800 kB
>>>>> CmaFree:         6602584 kB
>>>>>
>>>>> So the freed CMA memory is more than total..
>>>>>
>>>>> Also the the MemFree is more than mem total:
>>>>>
>>>>> -bash-4.3# cat /proc/meminfo
>>>>> MemTotal:       16342016 kB
>>>>> MemFree:        22367268 kB
>>>>> MemAvailable:   22370528 kB
>> [...]
>>>> I played with this a bit and can see the same problem. The sanity
>>>> check of CmaFree < CmaTotal generally triggers in
>>>> __move_zone_freepage_state in unset_migratetype_isolate.
>>>> This also seems to be present as far back as v4.0 which was the
>>>> first version to have the updated accounting from Joonsoo.
>>>> Were there known limitations with the new freepage accounting,
>>>> Joonsoo?
>>> I don't know. I also played with this and looks like there is
>>> accounting problem, however, for my case, number of free page is slightly less
>>> than total. I will take a look.
>>>
>>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
>>> look like your case.
>> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
>> did some other test:
> Thanks! Now, I can re-generate erronous situation you mentioned.
>
>>  - run with single thread with 100000 times, everything is fine.
>>
>>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>>    the same test with 100 multi-thread, then I got:
> [1] would not be sufficient to close this race.
>
> Try following things [A]. And, for more accurate test, I changed code a bit more
> to prevent kernel page allocation from cma area [B]. This will prevent kernel
> page allocation from cma area completely so we can focus cma_alloc/release race.
>
> Although, this is not correct fix, it could help that we can guess
> where the problem is.
>
> Thanks.
>
> [A]

I tested this solution [A], it can fix the problem, as you are posting a new patch, I will
test that one and leave [B] alone :)

Thanks
Hanjun

[toc] | [prev] | [next] | [standalone]

#1349903

On 2016/3/4 10:02, Joonsoo Kim wrote:
> On Thu, Mar 03, 2016 at 08:49:01PM +0800, Hanjun Guo wrote:
>> On 2016/3/3 15:42, Joonsoo Kim wrote:
>>> 2016-03-03 10:25 GMT+09:00 Laura Abbott <labbott@redhat.com>:
>>>> (cc -mm and Joonsoo Kim)
>>>>
>>>>
>>>> On 03/02/2016 05:52 AM, Hanjun Guo wrote:
>>>>> Hi,
>>>>>
>>>>> I came across a suspicious error for CMA stress test:
>>>>>
>>>>> Before the test, I got:
>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>> CmaTotal:         204800 kB
>>>>> CmaFree:          195044 kB
>>>>>
>>>>>
>>>>> After running the test:
>>>>> -bash-4.3# cat /proc/meminfo | grep Cma
>>>>> CmaTotal:         204800 kB
>>>>> CmaFree:         6602584 kB
>>>>>
>>>>> So the freed CMA memory is more than total..
>>>>>
>>>>> Also the the MemFree is more than mem total:
>>>>>
>>>>> -bash-4.3# cat /proc/meminfo
>>>>> MemTotal:       16342016 kB
>>>>> MemFree:        22367268 kB
>>>>> MemAvailable:   22370528 kB
>> [...]
>>>> I played with this a bit and can see the same problem. The sanity
>>>> check of CmaFree < CmaTotal generally triggers in
>>>> __move_zone_freepage_state in unset_migratetype_isolate.
>>>> This also seems to be present as far back as v4.0 which was the
>>>> first version to have the updated accounting from Joonsoo.
>>>> Were there known limitations with the new freepage accounting,
>>>> Joonsoo?
>>> I don't know. I also played with this and looks like there is
>>> accounting problem, however, for my case, number of free page is slightly less
>>> than total. I will take a look.
>>>
>>> Hanjun, could you tell me your malloc_size? I tested with 1 and it doesn't
>>> look like your case.
>> I tested with malloc_size with 2M, and it grows much bigger than 1M, also I
>> did some other test:
> Thanks! Now, I can re-generate erronous situation you mentioned.
>
>>  - run with single thread with 100000 times, everything is fine.
>>
>>  - I hack the cam_alloc() and free as below [1] to see if it's lock issue, with
>>    the same test with 100 multi-thread, then I got:
> [1] would not be sufficient to close this race.
>
> Try following things [A]. And, for more accurate test, I changed code a bit more
> to prevent kernel page allocation from cma area [B]. This will prevent kernel
> page allocation from cma area completely so we can focus cma_alloc/release race.
>
> Although, this is not correct fix, it could help that we can guess
> where the problem is.
>
> Thanks.
>
> [A]
> diff --git a/mm/cma.c b/mm/cma.c
> index c003274..43ed02d 100644
> --- a/mm/cma.c
> +++ b/mm/cma.c
> @@ -496,7 +496,9 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count)
>  
>         VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
>  
> +       mutex_lock(&cma_mutex);
>         free_contig_range(pfn, count);
> +       mutex_unlock(&cma_mutex);
>         cma_clear_bitmap(cma, pfn, count);
>         trace_cma_release(pfn, pages, count);
>  
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index c6c38ed..1ce8a59 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2192,7 +2192,8 @@ void free_hot_cold_page(struct page *page, bool cold)
>          * excessively into the page allocator
>          */
>         if (migratetype >= MIGRATE_PCPTYPES) {
> -               if (unlikely(is_migrate_isolate(migratetype))) {
> +               if (is_migrate_cma(migratetype) ||
> +                       unlikely(is_migrate_isolate(migratetype))) {
>                         free_one_page(zone, page, pfn, 0, migratetype);
>                         goto out;
>                 }

As I replied in previous email, the solution will fix the problem, the Cma freed memory and
system freed memory is in sane state after apply above patch.

I also tested this situation which only apply the code below:

        if (migratetype >= MIGRATE_PCPTYPES) {
-               if (unlikely(is_migrate_isolate(migratetype))) {
+               if (is_migrate_cma(migratetype) ||
+                       unlikely(is_migrate_isolate(migratetype))) {
                        free_one_page(zone, page, pfn, 0, migratetype);
                        goto out;
                }


This will not fix the problem, but will reduce the errorous freed number of memory,
hope this helps.

>
>
> [B]
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index f2dccf9..c6c38ed 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1493,6 +1493,7 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
>                                                                 int alloc_flags)
>  {
>         int i;
> +       bool cma = false;
>  
>         for (i = 0; i < (1 << order); i++) {
>                 struct page *p = page + i;
> @@ -1500,6 +1501,9 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
>                         return 1;
>         }
>  
> +       if (is_migrate_cma(get_pcppage_migratetype(page)))
> +               cma = true;
> +
>         set_page_private(page, 0);
>         set_page_refcounted(page);
>  
> @@ -1528,6 +1532,12 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
>         else
>                 clear_page_pfmemalloc(page);
>  
> +       if (cma) {
> +               page_ref_dec(page);

mm/page_alloc.c: In function ‘prep_new_page’:
mm/page_alloc.c:1407:3: error: implicit declaration of function ‘page_ref_dec’ [-Werror=implicit-function-declaration]
   page_ref_dec(page);
   ^

Typo?

Thanks
Hanjun

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web