Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1277919 > unrolled thread

[lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead

Back to article view | Back to linux.kernel

Contents

  [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead kernel test robot <ying.huang@linux.intel.com> - 2015-11-26 02:00 +0100
    Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Mel Gorman <mgorman@techsingularity.net> - 2015-11-26 14:30 +0100
      Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Rik van Riel <riel@redhat.com> - 2015-11-26 16:10 +0100
      Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-11-27 02:20 +0100
        Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Mel Gorman <mgorman@techsingularity.net> - 2015-11-27 11:10 +0100
          Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-11-30 03:20 +0100
            Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Michal Hocko <mhocko@kernel.org> - 2015-11-30 14:10 +0100

#1277919 — [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead

[Multipart message — attachments visible in raw view] — view raw

FYI, we noticed the below changes on

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd")

Note: the testing machine is a virtual machine with only 1G memory.

=========================================================================================
compiler/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-4.9/1HDD/16MB/xfs/1x/x86_64-rhel/16d/256fpd/32t/debian-x86_64-2015-02-07.cgz/fsyncBeforeClose/vm-vp-1G/60G/fsmark

commit: 
  016c13daa5c9e4827eca703e2f0621c131f2cca3
  d0164adc89f6bb374d304ffcc375c6d2652fe67d

016c13daa5c9e482 d0164adc89f6bb374d304ffcc3 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4          100%           4:4     last_state.fsmark.exit_code.143
           :4           50%           2:4     last_state.is_incomplete_run
           :4          100%           4:4     dmesg.Mem-Info
           :4          100%           4:4     dmesg.page_allocation_failure:order:#,mode
           :4          100%           4:4     dmesg.warn_alloc_failed+0x
      6327  23%     -93.5%     409.00  80%  proc-vmstat.allocstall
    173495  58%    -100.0%       0.00  -1%  proc-vmstat.compact_free_scanned
      4394  59%    -100.0%       0.00  -1%  proc-vmstat.compact_isolated
     10055  44%    -100.0%       0.00  -1%  proc-vmstat.compact_migrate_scanned
    443.25  13%     -99.7%       1.50 100%  proc-vmstat.kswapd_high_wmark_hit_quickly
     28950  12%     -91.4%       2502  81%  proc-vmstat.kswapd_low_wmark_hit_quickly
  15704144   0%     -91.1%    1402050  73%  proc-vmstat.nr_dirtied
     12851   0%     +26.3%      16235  18%  proc-vmstat.nr_dirty_background_threshold
     25704   0%     +26.3%      32471  18%  proc-vmstat.nr_dirty_threshold
      2882   0%   +1130.5%      35463  84%  proc-vmstat.nr_free_pages
  15693749   0%     -91.3%    1365065  75%  proc-vmstat.nr_written
  16289593   0%     -91.0%    1464689  72%  proc-vmstat.numa_hit
  16289593   0%     -91.0%    1464689  72%  proc-vmstat.numa_local
     30453  12%     -91.6%       2552  81%  proc-vmstat.pageoutrun
  16316641   0%     -91.0%    1468330  72%  proc-vmstat.pgalloc_dma32
    642889   5%     -90.5%      61326  56%  proc-vmstat.pgfault
  16218859   0%     -91.6%    1355797  78%  proc-vmstat.pgfree
     69.25  68%    -100.0%       0.00  -1%  proc-vmstat.pgmigrate_fail
      2066  58%    -100.0%       0.00  -1%  proc-vmstat.pgmigrate_success
  62849613   0%     -91.2%    5512004  74%  proc-vmstat.pgpgout
    417966  16%     -80.1%      82999  36%  proc-vmstat.pgscan_direct_dma32
  15259915   0%     -91.5%    1303209  76%  proc-vmstat.pgscan_kswapd_dma32
    360298  23%     -93.5%      23325  82%  proc-vmstat.pgsteal_direct_dma32
  15224912   0%     -91.7%    1270706  79%  proc-vmstat.pgsteal_kswapd_dma32
    236736   0%     -96.1%       9216 100%  proc-vmstat.slabs_scanned
    108153   0%     -98.0%       2154 100%  proc-vmstat.workingset_nodereclaim

vm-vp-1G: qemu-system-x86_64 -enable-kvm -cpu Nehalem
Memory: 1G

To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Ying Huang

[toc] | [next] | [standalone]

#1278211

On Thu, Nov 26, 2015 at 08:56:12AM +0800, kernel test robot wrote:
> FYI, we noticed the below changes on
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
> commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd")
> 
> Note: the testing machine is a virtual machine with only 1G memory.
> 

I'm not actually seeing any problem here. Is this a positive report or
am I missing something obvious?

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1278277

On 11/26/2015 08:25 AM, Mel Gorman wrote:
> On Thu, Nov 26, 2015 at 08:56:12AM +0800, kernel test robot wrote:
>> FYI, we noticed the below changes on
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
>> commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd")
>>
>> Note: the testing machine is a virtual machine with only 1G memory.
>>
>
> I'm not actually seeing any problem here. Is this a positive report or
> am I missing something obvious?

I've gotten several reports that could be either
positive or negative, but where I am not quite
sure how to interpret the results.

The tool seems to CC the maintainers of the code
that was changed, so I am hoping they will pipe
up when they see a problem.

Of course, that doesn't help in this case :)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1278489

Hi, Mel,

Mel Gorman <mgorman@techsingularity.net> writes:

> On Thu, Nov 26, 2015 at 08:56:12AM +0800, kernel test robot wrote:
>> FYI, we noticed the below changes on
>> 
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
>> commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc:
>> distinguish between being unable to sleep, unwilling to sleep and
>> avoiding waking kswapd")
>> 
>> Note: the testing machine is a virtual machine with only 1G memory.
>> 
>
> I'm not actually seeing any problem here. Is this a positive report or
> am I missing something obvious?

Sorry the email subject is generated automatically and I forget to
change it to some meaningful stuff before sending out.  From the testing
result, we found the commit make the OOM possibility increased from 0%
to 100% on this machine with small memory.  I also added proc-vmstat
information data too to help diagnose it.

Best Regards,
Huang, Ying
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1278696

On Fri, Nov 27, 2015 at 09:14:52AM +0800, Huang, Ying wrote:
> Hi, Mel,
> 
> Mel Gorman <mgorman@techsingularity.net> writes:
> 
> > On Thu, Nov 26, 2015 at 08:56:12AM +0800, kernel test robot wrote:
> >> FYI, we noticed the below changes on
> >> 
> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
> >> commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc:
> >> distinguish between being unable to sleep, unwilling to sleep and
> >> avoiding waking kswapd")
> >> 
> >> Note: the testing machine is a virtual machine with only 1G memory.
> >> 
> >
> > I'm not actually seeing any problem here. Is this a positive report or
> > am I missing something obvious?
> 
> Sorry the email subject is generated automatically and I forget to
> change it to some meaningful stuff before sending out.  From the testing
> result, we found the commit make the OOM possibility increased from 0%
> to 100% on this machine with small memory.  I also added proc-vmstat
> information data too to help diagnose it.
> 

There is no reference to OOM possibility in the email that I can see. Can
you give examples of the OOM messages that shows the problem sites? It was
suspected that there may be some callers that were accidentally depending
on access to emergency reserves. If so, either they need to be fixed (if
the case is extremely rare) or a small reserve will have to be created
for callers that are not high priority but still cannot reclaim.

Note that I'm travelling a lot over the next two weeks so I'll be slow to
respond but I will get to it.

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1279520

[Multipart message — attachments visible in raw view] — view raw

Mel Gorman <mgorman@techsingularity.net> writes:

> On Fri, Nov 27, 2015 at 09:14:52AM +0800, Huang, Ying wrote:
>> Hi, Mel,
>> 
>> Mel Gorman <mgorman@techsingularity.net> writes:
>> 
>> > On Thu, Nov 26, 2015 at 08:56:12AM +0800, kernel test robot wrote:
>> >> FYI, we noticed the below changes on
>> >> 
>> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
>> >> commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc:
>> >> distinguish between being unable to sleep, unwilling to sleep and
>> >> avoiding waking kswapd")
>> >> 
>> >> Note: the testing machine is a virtual machine with only 1G memory.
>> >> 
>> >
>> > I'm not actually seeing any problem here. Is this a positive report or
>> > am I missing something obvious?
>> 
>> Sorry the email subject is generated automatically and I forget to
>> change it to some meaningful stuff before sending out.  From the testing
>> result, we found the commit make the OOM possibility increased from 0%
>> to 100% on this machine with small memory.  I also added proc-vmstat
>> information data too to help diagnose it.
>> 
>
> There is no reference to OOM possibility in the email that I can see. Can
> you give examples of the OOM messages that shows the problem sites? It was
> suspected that there may be some callers that were accidentally depending
> on access to emergency reserves. If so, either they need to be fixed (if
> the case is extremely rare) or a small reserve will have to be created
> for callers that are not high priority but still cannot reclaim.
>
> Note that I'm travelling a lot over the next two weeks so I'll be slow to
> respond but I will get to it.

Here is the kernel log,  the full dmesg is attached too.  The OOM
occurs during fsmark testing.

Best Regards,
Huang, Ying

[   31.453514] kworker/u4:0: page allocation failure: order:0, mode:0x2200000
[   31.463570] CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
[   31.466115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
[   31.477146] Workqueue: writeback wb_workfn (flush-253:0)
[   31.481450]  0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
[   31.492582]  ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
[   31.507631]  ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000
[   31.510568] Call Trace:
[   31.511828]  [<ffffffff8140a142>] dump_stack+0x4b/0x69
[   31.513391]  [<ffffffff8117117b>] warn_alloc_failed+0xdb/0x140
[   31.523163]  [<ffffffff81174ec4>] __alloc_pages_nodemask+0x874/0xa60
[   31.524949]  [<ffffffff811bcb62>] alloc_pages_current+0x92/0x120
[   31.526659]  [<ffffffff811c73e4>] new_slab+0x3d4/0x480
[   31.536134]  [<ffffffff811c7c36>] __slab_alloc+0x376/0x470
[   31.537541]  [<ffffffff814e0ced>] ? alloc_indirect+0x1d/0x50
[   31.543268]  [<ffffffff81338221>] ? xfs_submit_ioend_bio+0x31/0x40
[   31.545104]  [<ffffffff814e0ced>] ? alloc_indirect+0x1d/0x50
[   31.546982]  [<ffffffff811c8e8d>] __kmalloc+0x20d/0x260
[   31.548334]  [<ffffffff814e0ced>] alloc_indirect+0x1d/0x50
[   31.549805]  [<ffffffff814e0fec>] virtqueue_add_sgs+0x2cc/0x3a0
[   31.555396]  [<ffffffff81573a30>] __virtblk_add_req+0xb0/0x1f0
[   31.556846]  [<ffffffff8117a121>] ? pagevec_lookup_tag+0x21/0x30
[   31.558318]  [<ffffffff813e5d72>] ? blk_rq_map_sg+0x1e2/0x4f0
[   31.563880]  [<ffffffff81573c82>] virtio_queue_rq+0x112/0x280
[   31.565307]  [<ffffffff813e9de7>] __blk_mq_run_hw_queue+0x1d7/0x370
[   31.571005]  [<ffffffff813e9bef>] blk_mq_run_hw_queue+0x9f/0xc0
[   31.572472]  [<ffffffff813eb10a>] blk_mq_insert_requests+0xfa/0x1a0
[   31.573982]  [<ffffffff813ebdb3>] blk_mq_flush_plug_list+0x123/0x140
[   31.583686]  [<ffffffff813e1777>] blk_flush_plug_list+0xa7/0x200
[   31.585138]  [<ffffffff813e1c49>] blk_finish_plug+0x29/0x40
[   31.586542]  [<ffffffff81215f85>] wb_writeback+0x185/0x2c0
[   31.592429]  [<ffffffff812166a5>] wb_workfn+0xf5/0x390
[   31.594037]  [<ffffffff81091297>] process_one_work+0x157/0x420
[   31.599804]  [<ffffffff81091ef9>] worker_thread+0x69/0x4a0
[   31.601484]  [<ffffffff81091e90>] ? rescuer_thread+0x380/0x380
[   31.611368]  [<ffffffff8109746f>] kthread+0xef/0x110
[   31.612953]  [<ffffffff81097380>] ? kthread_park+0x60/0x60
[   31.619418]  [<ffffffff818bce8f>] ret_from_fork+0x3f/0x70
[   31.621221]  [<ffffffff81097380>] ? kthread_park+0x60/0x60
[   31.635226] Mem-Info:
[   31.636569] active_anon:4942 inactive_anon:1643 isolated_anon:0
[   31.636569]  active_file:23196 inactive_file:110131 isolated_file:251
[   31.636569]  unevictable:92329 dirty:2865 writeback:1925 unstable:0
[   31.636569]  slab_reclaimable:10588 slab_unreclaimable:3390
[   31.636569]  mapped:2848 shmem:1687 pagetables:876 bounce:0
[   31.636569]  free:1932 free_pcp:218 free_cma:0
[   31.667096] Node 0 DMA free:3948kB min:60kB low:72kB high:88kB active_anon:264kB inactive_anon:128kB active_file:1544kB inactive_file:5296kB unevictable:3136kB isolated(anon):0kB isolated(file):236kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:440kB shmem:128kB slab_reclaimable:588kB slab_unreclaimable:304kB kernel_stack:112kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3376 all_unreclaimable? no
[   31.708140] lowmem_reserve[]: 0 972 972 972
[   31.710104] Node 0 DMA32 free:3780kB min:3824kB low:4780kB high:5736kB active_anon:19504kB inactive_anon:6444kB active_file:91240kB inactive_file:435228kB unevictable:366180kB isolated(anon):0kB isolated(file):768kB present:1032064kB managed:997532kB mlocked:0kB dirty:11460kB writeback:7700kB mapped:10952kB shmem:6620kB slab_reclaimable:41764kB slab_unreclaimable:13256kB kernel_stack:2752kB pagetables:3424kB unstable:0kB bounce:0kB free_pcp:872kB local_pcp:232kB free_cma:0kB writeback_tmp:0kB pages_scanned:140404 all_unreclaimable? no
[   31.743737] lowmem_reserve[]: 0 0 0 0
[   31.745320] Node 0 DMA: 7*4kB (UME) 2*8kB (UM) 2*16kB (ME) 1*32kB (E) 0*64kB 2*128kB (ME) 2*256kB (ME) 2*512kB (UM) 2*1024kB (ME) 0*2048kB 0*4096kB = 3948kB
[   31.757513] Node 0 DMA32: 1*4kB (U) 0*8kB 4*16kB (UME) 3*32kB (UE) 3*64kB (UM) 1*128kB (U) 1*256kB (U) 0*512kB 3*1024kB (UME) 0*2048kB 0*4096kB = 3812kB
[   31.766470] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[   31.772953] 227608 total pagecache pages
[   31.774127] 0 pages in swap cache
[   31.775428] Swap cache stats: add 0, delete 0, find 0/0
[   31.776785] Free swap  = 0kB
[   31.777799] Total swap = 0kB
[   31.779569] 262014 pages RAM
[   31.780584] 0 pages HighMem/MovableOnly
[   31.781744] 8654 pages reserved
[   31.790944] 0 pages hwpoisoned
[   31.792008] SLUB: Unable to allocate memory on node -1 (gfp=0x2080000)
[   31.793537]   cache: kmalloc-128, object size: 128, buffer size: 128, default order: 0, min order: 0
[   31.796088]   node 0: slabs: 27, objs: 864, free: 0

[toc] | [prev] | [next] | [standalone]

#1279850

[Let's CC Will - see the question at the end of the email please]

This seems to be a similar allocation failure reported
http://lkml.kernel.org/r/87oafjpnb1.fsf%40yhuang-dev.intel.com
where I failed to see the important point, more on that below.

On Mon 30-11-15 10:14:24, Huang, Ying wrote:
> Mel Gorman <mgorman@techsingularity.net> writes:
> 
> > On Fri, Nov 27, 2015 at 09:14:52AM +0800, Huang, Ying wrote:
> >> Hi, Mel,
> >> 
> >> Mel Gorman <mgorman@techsingularity.net> writes:
> >> 
> >> > On Thu, Nov 26, 2015 at 08:56:12AM +0800, kernel test robot wrote:
> >> >> FYI, we noticed the below changes on
> >> >> 
> >> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
> >> >> commit d0164adc89f6bb374d304ffcc375c6d2652fe67d ("mm, page_alloc:
> >> >> distinguish between being unable to sleep, unwilling to sleep and
> >> >> avoiding waking kswapd")
> >> >> 
> >> >> Note: the testing machine is a virtual machine with only 1G memory.
> >> >> 
> >> >
> >> > I'm not actually seeing any problem here. Is this a positive report or
> >> > am I missing something obvious?
> >> 
> >> Sorry the email subject is generated automatically and I forget to
> >> change it to some meaningful stuff before sending out.  From the testing
> >> result, we found the commit make the OOM possibility increased from 0%
> >> to 100% on this machine with small memory.  I also added proc-vmstat
> >> information data too to help diagnose it.
> >> 
> >
> > There is no reference to OOM possibility in the email that I can see. Can
> > you give examples of the OOM messages that shows the problem sites? It was
> > suspected that there may be some callers that were accidentally depending
> > on access to emergency reserves. If so, either they need to be fixed (if
> > the case is extremely rare) or a small reserve will have to be created
> > for callers that are not high priority but still cannot reclaim.

__virtblk_add_req calls
virtqueue_add_sgs(vq, sgs, num_out, num_in, vbr, GFP_ATOMIC)
  alloc_indirect(gfp)
    gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH)

So this is true __GFP_ATOMIC, we just drop __GFP_HIGH so it doesn't get
access to more reserves. It still does ALLOC_HARDER. So I think the real
issue is somewhere else when something should have triggered kswapd and
it doesn't do that anymore. I have tried to find that offender the last
time but didn't manage to find any.

Btw. I completely miss why b92b1b89a33c ("virtio: force vring
descriptors to be allocated from lowmem") had to clear __GFP_HIGH. Will
do you remember why you have dropped that flag as well?

Also I do not seem to find any user of alloc_indirect which would do
__GFP_HIGHMEM. All of them are either GFP_KERNEL or GFP_ATOMIC. So
either I am missing something or this is not really needed. Maybe the
situation was different back in 2012.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web