Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS

Path	csiph.com!news.mixmin.net!news.unit0.net!news.panservice.it!bofh.it!news.nic.it!robomod
From	Michal Hocko <mhocko@kernel.org>
Newsgroups	linux.kernel
Subject	Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS
Date	Wed, 01 Mar 2017 09:30:02 +0100
Message-ID	<tg7O2-jL-3@gated-at.bofh.it> (permalink)
References	<tfBOa-3yE-41@gated-at.bofh.it> <tfQNc-5z5-3@gated-at.bofh.it> <tfRgd-5JN-15@gated-at.bofh.it> <tfRgd-5JN-13@gated-at.bofh.it> <tfRJg-6ds-19@gated-at.bofh.it> <tfRJg-6ds-17@gated-at.bofh.it> <tfTi1-79a-3@gated-at.bofh.it> <tfTi2-79a-17@gated-at.bofh.it> <tfVMS-oA-19@gated-at.bofh.it>
X-Original-To	Robert Kudyba <rkudyba@fordham.edu>
MIME-Version	1.0
Content-Type	text/plain; charset=utf-8
Content-Disposition	inline
Content-Transfer-Encoding	8bit
User-Agent	Mutt/1.5.23 (2014-03-12)
Sender	robomod@news.nic.it
List-ID	<linux-kernel.vger.kernel.org>
X-Mailing-List	linux-kernel@vger.kernel.org
Approved	robomod@news.nic.it
Lines	44
Organization	linux.* mail to news gateway
X-Original-Cc	linux-kernel@vger.kernel.org
X-Original-Date	Wed, 1 Mar 2017 09:06:05 +0100
X-Original-Message-ID	<20170301080604.GB1124@dhcp22.suse.cz>
X-Original-References	<C16ACE34-A2F0-4A9F-BFBD-E369733A214F@fordham.edu> <20170228141520.GA28139@dhcp22.suse.cz> <40F07E96-7468-4355-B8EA-4B42F575ACAB@fordham.edu> <20170228144045.GD26792@dhcp22.suse.cz> <3E4C7821-A93D-4956-A0E0-730BEC67C9F0@fordham.edu> <20170228151535.GE26792@dhcp22.suse.cz> <63A3D887-EEDA-46D2-AB59-D5955FC3D23D@fordham.edu> <20170228165638.GA27726@dhcp22.suse.cz> <FD013725-A937-45AD-93AD-D79AF7E381BC@fordham.edu>
X-Original-Sender	linux-kernel-owner@vger.kernel.org
Xref	csiph.com linux.kernel:1590091

Show key headers only | View raw

On Tue 28-02-17 14:32:18, Robert Kudyba wrote:
> 
> > On Feb 28, 2017, at 11:56 AM, Michal Hocko <mhocko@kernel.org> wrote:
[...]
> >> Will do here’s a perf report:
> > 
> > this will not tell us much. Tracepoints have much better chance to tell
> > us how reclaim is progressing.
> 
> I have SystemTap configured are there any scripts in the
> SystemTap_Beginners_Guide.pdf that I can run to help? Sorry I’m
> brand new to tracepoints.
> 

I am not familiar with systemtap much. What I meant was to
mount -t tracefs none /trace
echo 1 > /trace/events/vmscan/enable

but

> I do see these “vmscan” from this command:
> stap -L 'kernel.trace("*")'|sort
> 
> kernel.trace("vmscan:mm_shrink_slab_end") $shr:struct shrinker* $nid:int $shrinker_retval:int $unused_scan_cnt:long int $new_scan_cnt:long int $total_scan:long int
> kernel.trace("vmscan:mm_shrink_slab_start") $shr:struct shrinker* $sc:struct shrink_control* $nr_objects_to_shrink:long int $pgs_scanned:long unsigned int $lru_pgs:long unsigned int $cache_items:long unsigned int $delta:long long unsigned int $total_scan:long unsigned int
> kernel.trace("vmscan:mm_vmscan_direct_reclaim_begin") $order:int $may_writepage:int $gfp_flags:gfp_t $classzone_idx:int
> kernel.trace("vmscan:mm_vmscan_direct_reclaim_end") $nr_reclaimed:long unsigned int
> kernel.trace("vmscan:mm_vmscan_kswapd_sleep") $nid:int
> kernel.trace("vmscan:mm_vmscan_kswapd_wake") $nid:int $zid:int $order:int
> kernel.trace("vmscan:mm_vmscan_lru_isolate") $classzone_idx:int $order:int $nr_requested:long unsigned int $nr_scanned:long unsigned int $nr_taken:long unsigned int $isolate_mode:isolate_mode_t $file:int
> kernel.trace("vmscan:mm_vmscan_lru_shrink_inactive") $nid:int $nr_scanned:long unsigned int $nr_reclaimed:long unsigned int $priority:int $file:int
> kernel.trace("vmscan:mm_vmscan_memcg_isolate") $classzone_idx:int $order:int $nr_requested:long unsigned int $nr_scanned:long unsigned int $nr_taken:long unsigned int $isolate_mode:isolate_mode_t $file:int
> kernel.trace("vmscan:mm_vmscan_memcg_reclaim_begin") $order:int $may_writepage:int $gfp_flags:gfp_t $classzone_idx:int
> kernel.trace("vmscan:mm_vmscan_memcg_reclaim_end") $nr_reclaimed:long unsigned int
> kernel.trace("vmscan:mm_vmscan_memcg_softlimit_reclaim_begin") $order:int $may_writepage:int $gfp_flags:gfp_t $classzone_idx:int
> kernel.trace("vmscan:mm_vmscan_memcg_softlimit_reclaim_end") $nr_reclaimed:long unsigned int
> kernel.trace("vmscan:mm_vmscan_wakeup_kswapd") $nid:int $zid:int $order:int
> kernel.trace("vmscan:mm_vmscan_writepage") $page:struct page*
 
this looks like it would achieve the same

-- 
Michal Hocko
SUSE Labs

Thread

rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Robert Kudyba <rkudyba@fordham.edu> - 2017-02-27 23:20 +0100
  Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-02-28 15:20 +0100
    Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-02-28 15:50 +0100
      Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-02-28 16:20 +0100
        Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Robert Kudyba <rkudyba@fordham.edu> - 2017-02-28 18:00 +0100
          Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-02-28 18:00 +0100
            Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Robert Kudyba <rkudyba@fordham.edu> - 2017-02-28 20:40 +0100
              Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-03-01 09:30 +0100
                Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-03-01 18:40 +0100
                Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Michal Hocko <mhocko@kernel.org> - 2017-03-01 20:30 +0100

csiph-web