Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.kernel > #1211957

[PATCH 3.12 82/82] rbd: fix copyup completion race

From Jiri Slaby <jslaby@suse.cz>
Newsgroups linux.kernel
Subject [PATCH 3.12 82/82] rbd: fix copyup completion race
Date 2015-08-24 11:20 +0200
Message-ID <q0W56-3om-51@gated-at.bofh.it> (permalink)
References <q0VVn-3cH-1@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw


From: Ilya Dryomov <idryomov@gmail.com>

3.12-stable review patch.  If anyone has any objections, please let me know.

===============

commit 2761713d35e370fd640b5781109f753066b746c4 upstream.

For write/discard obj_requests that involved a copyup method call, the
opcode of the first op is CEPH_OSD_OP_CALL and the ->callback is
rbd_img_obj_copyup_callback().  The latter frees copyup pages, sets
->xferred and delegates to rbd_img_obj_callback(), the "normal" image
object callback, for reporting to block layer and putting refs.

rbd_osd_req_callback() however treats CEPH_OSD_OP_CALL as a trivial op,
which means obj_request is marked done in rbd_osd_trivial_callback(),
*before* ->callback is invoked and rbd_img_obj_copyup_callback() has
a chance to run.  Marking obj_request done essentially means giving
rbd_img_obj_callback() a license to end it at any moment, so if another
obj_request from the same img_request is being completed concurrently,
rbd_img_obj_end_request() may very well be called on such prematurally
marked done request:

<obj_request-1/2 reply>
handle_reply()
  rbd_osd_req_callback()
    rbd_osd_trivial_callback()
    rbd_obj_request_complete()
    rbd_img_obj_copyup_callback()
    rbd_img_obj_callback()
                                    <obj_request-2/2 reply>
                                    handle_reply()
                                      rbd_osd_req_callback()
                                        rbd_osd_trivial_callback()
      for_each_obj_request(obj_request->img_request) {
        rbd_img_obj_end_request(obj_request-1/2)
        rbd_img_obj_end_request(obj_request-2/2) <--
      }

Calling rbd_img_obj_end_request() on such a request leads to trouble,
in particular because its ->xfferred is 0.  We report 0 to the block
layer with blk_update_request(), get back 1 for "this request has more
data in flight" and then trip on

    rbd_assert(more ^ (which == img_request->obj_request_count));

with rhs (which == ...) being 1 because rbd_img_obj_end_request() has
been called for both requests and lhs (more) being 1 because we haven't
got a chance to set ->xfferred in rbd_img_obj_copyup_callback() yet.

To fix this, leverage that rbd wants to call class methods in only two
cases: one is a generic method call wrapper (obj_request is standalone)
and the other is a copyup (obj_request is part of an img_request).  So
make a dedicated handler for CEPH_OSD_OP_CALL and directly invoke
rbd_img_obj_copyup_callback() from it if obj_request is part of an
img_request, similar to how CEPH_OSD_OP_READ handler invokes
rbd_img_obj_request_read_callback().

Since rbd_img_obj_copyup_callback() is now being called from the OSD
request callback (only), it is renamed to rbd_osd_copyup_callback().

Cc: Alex Elder <elder@linaro.org>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Alex Elder <elder@linaro.org>
[idryomov@gmail.com: backport to < 3.18: context]
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
---
 drivers/block/rbd.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 6aeaa28f94f0..63ff17fc23df 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -461,6 +461,7 @@ void rbd_warn(struct rbd_device *rbd_dev, const char *fmt, ...)
 #  define rbd_assert(expr)	((void) 0)
 #endif /* !RBD_DEBUG */
 
+static void rbd_osd_copyup_callback(struct rbd_obj_request *obj_request);
 static int rbd_img_obj_request_submit(struct rbd_obj_request *obj_request);
 static void rbd_img_parent_read(struct rbd_obj_request *obj_request);
 static void rbd_dev_remove_parent(struct rbd_device *rbd_dev);
@@ -1664,6 +1665,16 @@ static void rbd_osd_stat_callback(struct rbd_obj_request *obj_request)
 	obj_request_done_set(obj_request);
 }
 
+static void rbd_osd_call_callback(struct rbd_obj_request *obj_request)
+{
+	dout("%s: obj %p\n", __func__, obj_request);
+
+	if (obj_request_img_data_test(obj_request))
+		rbd_osd_copyup_callback(obj_request);
+	else
+		obj_request_done_set(obj_request);
+}
+
 static void rbd_osd_req_callback(struct ceph_osd_request *osd_req,
 				struct ceph_msg *msg)
 {
@@ -1702,6 +1713,8 @@ static void rbd_osd_req_callback(struct ceph_osd_request *osd_req,
 		rbd_osd_stat_callback(obj_request);
 		break;
 	case CEPH_OSD_OP_CALL:
+		rbd_osd_call_callback(obj_request);
+		break;
 	case CEPH_OSD_OP_NOTIFY_ACK:
 	case CEPH_OSD_OP_WATCH:
 		rbd_osd_trivial_callback(obj_request);
@@ -2293,13 +2306,15 @@ out_unwind:
 }
 
 static void
-rbd_img_obj_copyup_callback(struct rbd_obj_request *obj_request)
+rbd_osd_copyup_callback(struct rbd_obj_request *obj_request)
 {
 	struct rbd_img_request *img_request;
 	struct rbd_device *rbd_dev;
 	struct page **pages;
 	u32 page_count;
 
+	dout("%s: obj %p\n", __func__, obj_request);
+
 	rbd_assert(obj_request->type == OBJ_REQUEST_BIO);
 	rbd_assert(obj_request_img_data_test(obj_request));
 	img_request = obj_request->img_request;
@@ -2325,9 +2340,7 @@ rbd_img_obj_copyup_callback(struct rbd_obj_request *obj_request)
 	if (!obj_request->result)
 		obj_request->xferred = obj_request->length;
 
-	/* Finish up with the normal image object callback */
-
-	rbd_img_obj_callback(obj_request);
+	obj_request_done_set(obj_request);
 }
 
 static void
@@ -2424,7 +2437,6 @@ rbd_img_obj_parent_read_full_callback(struct rbd_img_request *img_request)
 
 	/* All set, send it off. */
 
-	orig_request->callback = rbd_img_obj_copyup_callback;
 	osdc = &rbd_dev->rbd_client->client->osdc;
 	img_result = rbd_obj_request_submit(osdc, orig_request);
 	if (!img_result)
-- 
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | NextPrevious in thread | Next in thread | Find similar | Unroll thread


Thread

[PATCH 3.12 01/82] efi: fix 32bit kernel boot failed problem using efi Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:10 +0200
  [PATCH 3.12 67/82] signalfd: fix information leak in signalfd_copyinfo Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 54/82] ipc: modify message queue accounting to not take kernel data structures into account Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 75/82] mm/hwpoison: fix page refcount of unknown non LRU page Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 64/82] x86/ldt: Correct LDT access in single stepping logic Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 44/82] ipmi: fix timeout calculation when bmc is disconnected Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 07/82] ARC: make sure instruction_pointer() returns unsigned value Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 59/82] x86/nmi/64: Switch stacks on userspace NMI entry Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 62/82] rcu: Move lockless_dereference() out of rcupdate.h Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 55/82] ocfs2: fix BUG in ocfs2_downconvert_thread_do_work() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 60/82] arch: Introduce smp_load_acquire(), smp_store_release() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 57/82] x86/nmi: Enable nested do_nmi() handling for 64-bit kernels Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 71/82] md/bitmap: return an error when bitmap superblock is corrupt. Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 76/82] xen-blkfront: don't add indirect pages to list when !feature_persistent Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 45/82] sparc64: Fix userspace FPU register corruptions. Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 74/82] ipc,sem: fix use after free on IPC_RMID after a task using same semaphore set exits Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 05/82] freeing unlinked file indefinitely delayed Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 66/82] x86/ldt: Further fix FPU emulation Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 69/82] signal: fix information leak in copy_siginfo_from_user32 Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 80/82] EDAC, ppc4xx: Access mci->csrows array elements properly Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 56/82] md/raid1: extend spinlock to protect raid1_end_read_request against inconsistencies Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 77/82] perf: Fix fasync handling on inherited events Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 63/82] x86/ldt: Make modify_ldt synchronous Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 82/82] rbd: fix copyup completion race Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 78/82] dm thin metadata: delete btrees when releasing metadata snapshot Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 73/82] ipc/sem.c: update/correct memory barriers Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 65/82] x86/ldt: Correct FPU emulation access to LDT Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 68/82] signal: fix information leak in copy_siginfo_to_user Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 81/82] drm/radeon: add new OLAND pci id Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 58/82] x86/nmi/64: Remove asm code that saves CR2 Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 79/82] localmodconfig: Use Kbuild files too Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 61/82] rcu: Provide counterpart to rcu_dereference() for non-RCU situations Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 72/82] mm, vmscan: Do not wait for page writeback for GFP_NOFS allocations Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 46/82] md: use kzalloc() when bitmap is disabled Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 70/82] path_openat(): fix double fput() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:20 +0200
  [PATCH 3.12 39/82] ipr: Fix invalid array indexing for HRRQ Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 38/82] ipr: Fix incorrect trace indexing Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 51/82] ARM: OMAP2+: hwmod: Fix _wait_target_ready() for hwmods without sysc Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 32/82] ARM: realview: fix sparsemem build Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 31/82] hwrng: via-rng - Mark device ID table as __maybe_unused Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 10/82] ALSA: usb-audio: add dB range mapping for some devices Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 33/82] MIPS: Fix sched_getaffinity with MT FPAFF enabled Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 43/82] ima: extend "mask" policy matching support Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 41/82] USB: sierra: add 1199:68AB device ID Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 30/82] 3w-xxxx: fix mis-aligned struct accesses Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 40/82] xhci: fix off by one error in TRB DMA address boundary check Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 53/82] ALSA: hda - fix cs4210_spdif_automute() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 08/82] genirq: Prevent resend to interrupts marked IRQ_NESTED_THREAD Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 52/82] iscsi-target: Fix iscsit_start_kthreads failure OOPs Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 27/82] vhost: actually track log eventfd file Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 35/82] fsnotify: fix oops in fsnotify_clear_marks_by_group_flags() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 37/82] ipr: Fix locking for unit attention handling Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 36/82] drm/radeon/combios: add some validation of lvds values Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 47/82] ASoC: pcm1681: Fix setting de-emphasis sampling rate selection Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 50/82] crypto: ixp4xx - Remove bogus BUG_ON on scattered dst buffer Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 29/82] iscsi-target: Fix iser explicit logout TX kthread leak Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 48/82] x86/xen: Probe target addresses in set_aliased_prot() before the hypercall Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 49/82] xen/gntdevt: Fix race condition in gntdev_release() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
    Re: [PATCH 3.12 49/82] xen/gntdevt: Fix race condition in  gntdev_release() Luis Henriques <luis.henriques@canonical.com> - 2015-08-25 13:40 +0200
      Re: [PATCH 3.12 49/82] xen/gntdevt: Fix race condition in  gntdev_release() Marek Marczykowski-Górecki   <marmarek@invisiblethingslab.com> - 2015-08-25 14:00 +0200
        Re: [PATCH 3.12 49/82] xen/gntdevt: Fix race condition in  gntdev_release() Jiri Slaby <jslaby@suse.cz> - 2015-08-25 15:20 +0200
          Re: [PATCH 3.12 49/82] xen/gntdevt: Fix race condition in  gntdev_release() Marek Marczykowski-Górecki   <marmarek@invisiblethingslab.com> - 2015-08-25 16:10 +0200
            Re: [PATCH 3.12 49/82] xen/gntdevt: Fix race condition in  gntdev_release() Jiri Slaby <jslaby@suse.cz> - 2015-08-27 10:10 +0200
  [PATCH 3.12 42/82] ima: add support for new "euid" policy condition Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 34/82] MIPS: Make set_pte() SMP safe. Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:30 +0200
  [PATCH 3.12 02/82] futex: Fix a race condition between REQUEUE_PI and task death Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 28/82] iscsi-target: Fix use-after-free during TPG session shutdown Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 09/82] ALSA: usb-audio: Add MIDI support for Steinberg MI2/MI4 Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 15/82] mmc: sdhci-pxav3: fix platform_data is not initialized Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 19/82] blkcg: fix gendisk reference leak in blkg_conf_prep() Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 26/82] rds: rds_ib_device.refcount overflow Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 23/82] xhci: report U3 when link is in resume state Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 03/82] HID: usbhid: add Chicony/Pixart usb optical mouse that needs QUIRK_ALWAYS_POLL Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 18/82] Input: usbtouchscreen - avoid unresponsive TSC-30 touch screen Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 25/82] xhci: do not report PLC when link is in internal resume state Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 22/82] xhci: Calculate old endpoints correctly on device reset Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 24/82] xhci: prevent bus_suspend if SS port resuming in phase 1 Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 06/82] s390/sclp: clear upper register halves in _sclp_print_early Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 17/82] tile: use free_bootmem_late() for initrd Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 04/82] mm: avoid setting up anonymous pages into file mapping Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 16/82] md/raid1: fix test for 'was read error from last working device'. Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 20/82] ata: pmp: add quirk for Marvell 4140 SATA PMP Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 11/82] ALSA: hda - Fix MacBook Pro 5,2 quirk Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 13/82] mac80211: clear subdir_stations when removing debugfs Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 21/82] usb-storage: ignore ZTE MF 823 card reader in mode 0x1225 Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 14/82] mmc: sdhci-esdhc: Make 8BIT bus work Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200
  [PATCH 3.12 12/82] st: null pointer dereference panic caused by use after kref_put by st_open Jiri Slaby <jslaby@suse.cz> - 2015-08-24 11:40 +0200

csiph-web