Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1736230

Re: [PATCH 0/6] More graceful flusher thread memory reclaim wakeup

Show all headers | View raw

[ Fixed up CC list. John, you're sending email with
  From: John Stoffel <john@quad.stoffel.home> ]

On Wed, Sep 20, 2017 at 01:32:25PM -0600, Jens Axboe wrote:
> On 09/20/2017 01:29 PM, John Stoffel wrote:
> > On Tue, Sep 19, 2017 at 01:53:01PM -0600, Jens Axboe wrote:
> >> We've had some issues with writeback in presence of memory reclaim
> >> at Facebook, and this patch set attempts to fix it up. The real
> >> functional change is the last patch in the series, the first 5 are
> >> prep and cleanup patches.
> >>
> >> The basic idea is that we have callers that call
> >> wakeup_flusher_threads() with nr_pages == 0. This means 'writeback
> >> everything'. For memory reclaim situations, we can end up queuing
> >> a TON of these kinds of writeback units. This can cause softlockups
> >> and further memory issues, since we allocate huge amounts of
> >> struct wb_writeback_work to handle this writeback. Handle this
> >> situation more gracefully.
> > 
> > This looks nice, but do you have any numbers to show how this improves
> > things?  I read the patches, but I'm not strong enough to comment on
> > them at all.  But I am interested in how this improves writeback under
> > pressure, if at all.
> 
> Writeback should be about the same, it's mostly about preventing
> softlockups and excessive memory usage, under conditions where we are
> actively trying to reclaim/clean memory. It was bad enough to cause
> softlockups for writeback work processing, while the pending writeback
> work units grew to insane lengths.

In numbers, we have seen situations where we had 600 million writeback
work items queued up from reclaim under pressure. That's 35G worth of
work descriptors, and the machine was struggling to remain responsive
due to a lack of memory.

Once writeback against all outstanding dirty pages has been requested,
there really isn't a need to queue even a second work item; the job is
already being performed. We can queue the next one when it completes.

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

[PATCH 0/6] More graceful flusher thread memory reclaim wakeup Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:00 +0200
  [PATCH 3/6] page-writeback: pass in '0' for nr_pages writeback in laptop mode Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:00 +0200
    Re: [PATCH 3/6] page-writeback: pass in '0' for nr_pages writeback  in laptop mode Johannes Weiner <hannes@cmpxchg.org> - 2017-09-19 22:10 +0200
    Re: [PATCH 3/6] page-writeback: pass in '0' for nr_pages writeback  in laptop mode Jan Kara <jack@suse.cz> - 2017-09-20 16:40 +0200
      Re: [PATCH 3/6] page-writeback: pass in '0' for nr_pages writeback in  laptop mode Jens Axboe <axboe@kernel.dk> - 2017-09-20 17:30 +0200
  [PATCH 2/6] fs-writeback: provide a wakeup_flusher_threads_bdi() Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:00 +0200
    Re: [PATCH 2/6] fs-writeback: provide a wakeup_flusher_threads_bdi() Johannes Weiner <hannes@cmpxchg.org> - 2017-09-19 22:10 +0200
    Re: [PATCH 2/6] fs-writeback: provide a wakeup_flusher_threads_bdi() Jan Kara <jack@suse.cz> - 2017-09-20 16:30 +0200
  [PATCH 5/6] fs-writeback: move nr_pages == 0 logic to one location Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:00 +0200
    Re: [PATCH 5/6] fs-writeback: move nr_pages == 0 logic to one  location Johannes Weiner <hannes@cmpxchg.org> - 2017-09-19 22:10 +0200
    Re: [PATCH 5/6] fs-writeback: move nr_pages == 0 logic to one  location Jan Kara <jack@suse.cz> - 2017-09-20 16:50 +0200
      Re: [PATCH 5/6] fs-writeback: move nr_pages == 0 logic to one  location Jens Axboe <axboe@kernel.dk> - 2017-09-20 17:10 +0200
        Re: [PATCH 5/6] fs-writeback: move nr_pages == 0 logic to one  location Jan Kara <jack@suse.cz> - 2017-09-20 17:40 +0200
          Re: [PATCH 5/6] fs-writeback: move nr_pages == 0 logic to one  location Jens Axboe <axboe@kernel.dk> - 2017-09-20 17:50 +0200
  [PATCH 1/6] buffer: cleanup free_more_memory() flusher wakeup Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:00 +0200
    Re: [PATCH 1/6] buffer: cleanup free_more_memory() flusher wakeup Johannes Weiner <hannes@cmpxchg.org> - 2017-09-19 22:10 +0200
    Re: [PATCH 1/6] buffer: cleanup free_more_memory() flusher wakeup Jan Kara <jack@suse.cz> - 2017-09-20 16:20 +0200
      Re: [PATCH 1/6] buffer: cleanup free_more_memory() flusher wakeup Jens Axboe <axboe@kernel.dk> - 2017-09-20 17:20 +0200
  [PATCH 6/6] fs-writeback: only allow one inflight and pending !nr_pages flush Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:00 +0200
    Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Johannes Weiner <hannes@cmpxchg.org> - 2017-09-19 22:20 +0200
      Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Jens Axboe <axboe@kernel.dk> - 2017-09-19 22:40 +0200
    Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Jens Axboe <axboe@kernel.dk> - 2017-09-20 04:00 +0200
    Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Amir Goldstein <amir73il@gmail.com> - 2017-09-20 05:20 +0200
      Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Jens Axboe <axboe@kernel.dk> - 2017-09-20 06:20 +0200
        Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Amir Goldstein <amir73il@gmail.com> - 2017-09-20 08:10 +0200
          Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Jens Axboe <axboe@kernel.dk> - 2017-09-20 14:40 +0200
        Re: [PATCH 6/6] fs-writeback: only allow one inflight and pending  !nr_pages flush Jan Kara <jack@suse.cz> - 2017-09-20 16:50 +0200
  Re: [PATCH 0/6] More graceful flusher thread memory reclaim wakeup John Stoffel <john@quad.stoffel.home> - 2017-09-20 21:40 +0200
    Re: [PATCH 0/6] More graceful flusher thread memory reclaim wakeup Jens Axboe <axboe@kernel.dk> - 2017-09-20 21:40 +0200
      Re: [PATCH 0/6] More graceful flusher thread memory reclaim wakeup Johannes Weiner <hannes@cmpxchg.org> - 2017-09-21 01:20 +0200

csiph-web