Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1608801

[PATCH 4.4 22/30] md/raid1/10: fix potential deadlock

Show all headers | View raw

4.4-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Shaohua Li <shli@fb.com>

commit 61eb2b43b99ebdc9bc6bc83d9792257b243e7cb3 upstream.

Neil Brown pointed out a potential deadlock in raid 10 code with
bio_split/chain. The raid1 code could have the same issue, but recent
barrier rework makes it less likely to happen. The deadlock happens in
below sequence:

1. generic_make_request(bio), this will set current->bio_list
2. raid10_make_request will split bio to bio1 and bio2
3. __make_request(bio1), wait_barrer, add underlayer disk bio to
current->bio_list
4. __make_request(bio2), wait_barrer

If raise_barrier happens between 3 & 4, since wait_barrier runs at 3,
raise_barrier waits for IO completion from 3. And since raise_barrier
sets barrier, 4 waits for raise_barrier. But IO from 3 can't be
dispatched because raid10_make_request() doesn't finished yet.

The solution is to adjust the IO ordering. Quotes from Neil:
"
It is much safer to:

    if (need to split) {
        split = bio_split(bio, ...)
        bio_chain(...)
        make_request_fn(split);
        generic_make_request(bio);
   } else
        make_request_fn(mddev, bio);

This way we first process the initial section of the bio (in 'split')
which will queue some requests to the underlying devices.  These
requests will be queued in generic_make_request.
Then we queue the remainder of the bio, which will be added to the end
of the generic_make_request queue.
Then we return.
generic_make_request() will pop the lower-level device requests off the
queue and handle them first.  Then it will process the remainder
of the original bio once the first section has been fully processed.
"

Note, this only happens in read path. In write path, the bio is flushed to
underlaying disks either by blk flush (from schedule) or offladed to raid1/10d.
It's queued in current->bio_list.

Cc: Coly Li <colyli@suse.de>
Suggested-by: NeilBrown <neilb@suse.com>
Reviewed-by: Jack Wang <jinpu.wang@profitbricks.com>
Signed-off-by: Shaohua Li <shli@fb.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 drivers/md/raid10.c |   18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1477,7 +1477,25 @@ static void make_request(struct mddev *m
 			split = bio;
 		}
 
+		/*
+		 * If a bio is splitted, the first part of bio will pass
+		 * barrier but the bio is queued in current->bio_list (see
+		 * generic_make_request). If there is a raise_barrier() called
+		 * here, the second part of bio can't pass barrier. But since
+		 * the first part bio isn't dispatched to underlaying disks
+		 * yet, the barrier is never released, hence raise_barrier will
+		 * alays wait. We have a deadlock.
+		 * Note, this only happens in read path. For write path, the
+		 * first part of bio is dispatched in a schedule() call
+		 * (because of blk plug) or offloaded to raid10d.
+		 * Quitting from the function immediately can change the bio
+		 * order queued in bio_list and avoid the deadlock.
+		 */
 		__make_request(mddev, split);
+		if (split != bio && bio_data_dir(bio) == READ) {
+			generic_make_request(bio);
+			break;
+		}
 	} while (split != bio);
 
 	/* In case raid10d snuck in to freeze_array */

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH 4.4 00/30] 4.4.57-stable review Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:10 +0100
  [PATCH 4.4 16/30] hv_netvsc: use skb_get_hash() instead of a homegrown implementation Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 23/30] target/pscsi: Fix TYPE_TAPE + TYPE_MEDIMUM_CHANGER export Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 21/30] powerpc/boot: Fix zImage TOC alignment Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 12/30] KVM: PPC: Book3S PR: Fix illegal opcode emulation Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 17/30] kernek/fork.c: allocate idle task for a CPU always on its local node Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 19/30] perf/core: Fix event inheritance on fork() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 01/30] usb: core: hub: hub_port_init lock controller instead of bus Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 22/30] md/raid1/10: fix potential deadlock Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 02/30] USB: dont free bandwidth_mutex too early Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 13/30] s390/pci: fix use after free in dma_init Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 15/30] tpm_tis: Use devm_free_irq not free_irq Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 24/30] scsi: lpfc: Add shutdown method for kexec Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  [PATCH 4.4 20/30] cpufreq: Fix and clean up show_cpuinfo_cur_freq() Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-03-24 19:20 +0100
  Re: [PATCH 4.4 00/30] 4.4.57-stable review Shuah Khan <shuahkh@osg.samsung.com> - 2017-03-25 01:10 +0100
  Re: [PATCH 4.4 00/30] 4.4.57-stable review Guenter Roeck <linux@roeck-us.net> - 2017-03-25 05:20 +0100
  Re: [PATCH 4.4 00/30] 4.4.57-stable review Guenter Roeck <linux@roeck-us.net> - 2017-03-25 05:20 +0100
    Re: [PATCH 4.4 00/30] 4.4.57-stable review Alexandre Belloni <alexandre.belloni@free-electrons.com> - 2017-03-25 12:30 +0100
      Re: [PATCH 4.4 00/30] 4.4.57-stable review Alexandre Belloni <alexandre.belloni@free-electrons.com> - 2017-03-25 12:40 +0100

csiph-web