Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1381523
| From | lizf@kernel.org |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH 3.4 91/92] splice: sendfile() at once fails for big files |
| Date | 2016-04-18 13:00 +0200 |
| Message-ID | <rpf4n-46M-47@gated-at.bofh.it> (permalink) |
| References | <rpeUF-424-3@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
From: Christophe Leroy <christophe.leroy@c-s.fr>
3.4.112-rc1 review patch. If anyone has any objections, please let me know.
------------------
commit 0ff28d9f4674d781e492bcff6f32f0fe48cf0fed upstream.
Using sendfile with below small program to get MD5 sums of some files,
it appear that big files (over 64kbytes with 4k pages system) get a
wrong MD5 sum while small files get the correct sum.
This program uses sendfile() to send a file to an AF_ALG socket
for hashing.
/* md5sum2.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <linux/if_alg.h>
int main(int argc, char **argv)
{
int sk = socket(AF_ALG, SOCK_SEQPACKET, 0);
struct stat st;
struct sockaddr_alg sa = {
.salg_family = AF_ALG,
.salg_type = "hash",
.salg_name = "md5",
};
int n;
bind(sk, (struct sockaddr*)&sa, sizeof(sa));
for (n = 1; n < argc; n++) {
int size;
int offset = 0;
char buf[4096];
int fd;
int sko;
int i;
fd = open(argv[n], O_RDONLY);
sko = accept(sk, NULL, 0);
fstat(fd, &st);
size = st.st_size;
sendfile(sko, fd, &offset, size);
size = read(sko, buf, sizeof(buf));
for (i = 0; i < size; i++)
printf("%2.2x", buf[i]);
printf(" %s\n", argv[n]);
close(fd);
close(sko);
}
exit(0);
}
Test below is done using official linux patch files. First result is
with a software based md5sum. Second result is with the program above.
root@vgoip:~# ls -l patch-3.6.*
-rw-r--r-- 1 root root 64011 Aug 24 12:01 patch-3.6.2.gz
-rw-r--r-- 1 root root 94131 Aug 24 12:01 patch-3.6.3.gz
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
5fd77b24e68bb24dcc72d6e57c64790e patch-3.6.3.gz
After investivation, it appears that sendfile() sends the files by blocks
of 64kbytes (16 times PAGE_SIZE). The problem is that at the end of each
block, the SPLICE_F_MORE flag is missing, therefore the hashing operation
is reset as if it was the end of the file.
This patch adds SPLICE_F_MORE to the flags when more data is pending.
With the patch applied, we get the correct sums:
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Jens Axboe <axboe@fb.com>
Cc: Ben Hutchings <ben@decadent.org.uk>
Signed-off-by: Zefan Li <lizefan@huawei.com>
---
fs/splice.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/fs/splice.c b/fs/splice.c
index 67c5210..2864177 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -1165,7 +1165,7 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
long ret, bytes;
umode_t i_mode;
size_t len;
- int i, flags;
+ int i, flags, more;
/*
* We require the input being a regular file, as we don't want to
@@ -1208,6 +1208,7 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
* Don't block on output, we have to drain the direct pipe.
*/
sd->flags &= ~SPLICE_F_NONBLOCK;
+ more = sd->flags & SPLICE_F_MORE;
while (len) {
size_t read_len;
@@ -1221,6 +1222,15 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
sd->total_len = read_len;
/*
+ * If more data is pending, set SPLICE_F_MORE
+ * If this is the last data and SPLICE_F_MORE was not set
+ * initially, clears it.
+ */
+ if (read_len < len)
+ sd->flags |= SPLICE_F_MORE;
+ else if (!more)
+ sd->flags &= ~SPLICE_F_MORE;
+ /*
* NOTE: nonblocking mode only applies to the input. We
* must not do the output in nonblocking mode as then we
* could get stuck data in the internal pipe:
--
1.9.1
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH 3.4 00/92] 3.4.112-rc1 review lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 41/92] iwlwifi: dvm: fix D3 firmware PN programming lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 10/92] devres: fix devres_get() lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 19/92] SUNRPC: xs_reset_transport must mark the connection as disconnected lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 29/92] scsi_dh: fix randconfig build error lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 45/92] md/raid10: ensure device failure recorded before write request returns. lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 42/92] sched/core: Fix TASK_DEAD race in finish_task_switch() lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 16/92] DRM - radeon: Don't link train DisplayPort on HPD until we get the dpcd lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 24/92] hpfs: update ctime and mtime on directory modification lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 31/92] powerpc/MSI: Fix race condition in tearing down MSI interrupts lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 26/92] fs: create and use seq_show_option for escaping lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 22/92] Add radeon suspend/resume quirk for HP Compaq dc5750. lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 46/92] md/raid10: don't clear bitmap bit when bad-block-list write fails. lizf@kernel.org - 2016-04-18 12:50 +0200
[PATCH 3.4 73/92] tty: fix stall caused by missing memory barrier in drivers/tty/n_tty.c lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 67/92] x86/process: Add proper bound checks in 64bit get_wchan() lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 51/92] spi: Fix documentation of spi_alloc_master() lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 64/92] x86/xen: Do not clip xen_e820_map to xen_e820_map_entries when sanitizing map lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 69/92] mm: hugetlbfs: skip shared VMAs when unmapping private pages to satisfy a fault lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 85/92] mm: make sendfile(2) killable lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 56/92] usb: Use the USB_SS_MULT() macro to get the burst multiplier. lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 71/92] USB: Add reset-resume quirk for two Plantronics usb headphones. lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 76/92] xen-blkfront: check for null drvdata in blkback_changed (XenbusStateClosing) lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 72/92] usb: Add device quirk for Logitech PTZ cameras lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 68/92] genirq: Fix race in register_irq_proc() lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 92/92] x86/iopl/64: Properly context-switch IOPL on Xen PV lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 91/92] splice: sendfile() at once fails for big files lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 82/92] ASoC: wm8904: Correct number of EQ registers lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 90/92] pipe: Fix buffer offset after partially failed read lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 89/92] usb: Use the USB_SS_MULT() macro to decode burst multiplier for log message lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 80/92] xhci: Add spurious wakeup quirk for LynxPoint-LP controllers lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 88/92] raid1: include bio_end_io_list in nr_queued to prevent freeze_array hang lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 83/92] iommu/amd: Don't clear DTE flags when modifying it lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 78/92] iommu/vt-d: fix range computation when making room for large pages lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 58/92] usb: xhci: Clear XHCI_STATE_DYING on start lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 52/92] btrfs: skip waiting on ordered range for special files lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 77/92] crypto: ahash - ensure statesize is non-zero lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 87/92] mvsas: Fix NULL pointer dereference in mvs_slot_task_free lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 86/92] dm btree: fix leak of bufio-backed block in btree_split_beneath error path lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 74/92] drivers/tty: require read access for controlling terminal lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 60/92] cifs: use server timestamp for ntlmv2 authentication lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 75/92] ALSA: synth: Fix conflicting OSS device registration on AWE32 lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 81/92] crypto: api - Only abort operations on fatal signal lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 84/92] drm/nouveau/gem: return only valid domain when there's only one lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 61/92] ocfs2/dlm: fix deadlock when dispatch assert master lizf@kernel.org - 2016-04-18 13:00 +0200
Re: [PATCH 3.4 61/92] ocfs2/dlm: fix deadlock when dispatch assert master Joseph Qi <joseph.qi@huawei.com> - 2016-04-18 13:40 +0200
Re: [PATCH 3.4 61/92] ocfs2/dlm: fix deadlock when dispatch assert master Zefan Li <lizefan@huawei.com> - 2016-04-19 02:20 +0200
[PATCH 3.4 79/92] xhci: handle no ping response error properly lizf@kernel.org - 2016-04-18 13:00 +0200
[PATCH 3.4 57/92] xhci: give command abortion one more chance before killing xhci lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 50/92] spi: spi-pxa2xx: Check status register to determine if SSSR_TINT is disabled lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 48/92] md/raid1: don't clear bitmap bit when bad-block-list write fails. lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 43/92] IB/cm: Fix rb-tree duplicate free and use-after-free lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 49/92] drm: crtc: integer overflow in drm_property_create_blob() lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 55/92] KVM: x86: trap AMD MSRs for the TSeg base and mask lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 63/92] m68k: Define asmlinkage_protect lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 54/92] regmap: debugfs: Don't bother actually printing when calculating max length lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 66/92] UBI: return ENOSPC if no enough space available lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 59/92] xhci: change xhci 1.0 only restrictions to support xhci 1.1 lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 53/92] regmap: debugfs: Ensure we don't underflow when printing access masks lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 65/92] UBI: Validate data_size lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 62/92] ath9k: declare required extra tx headroom lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 70/92] clocksource: Fix abs() usage w/ 64bit values lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 47/92] md/raid1: ensure device failure recorded before write request returns. lizf@kernel.org - 2016-04-18 13:10 +0200
[PATCH 3.4 09/92] auxdisplay: ks0108: fix refcount lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 25/92] crypto: ghash-clmulni: specify context size for ghash async algorithm lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 20/92] IB/mlx4: Use correct SL on AH query under RoCE lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 21/92] IB/uverbs: Fix race between ib_uverbs_open and remove_one lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 30/92] ARM: 8429/1: disable GCC SRA optimization lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 33/92] ARM: 7880/1: Clear the IT state independent of the Thumb-2 mode lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 13/92] usb: host: ehci-sys: delete useless bus_to_hcd conversion lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 40/92] md/raid0: apply base queue limits *before* disk_stack_limits lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 15/92] eCryptfs: Invalidate dcache entries when lower i_nlink is zero lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 38/92] ASoC: fix broken pxa SoC support lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 06/92] powerpc/rtas: Introduce rtas_get_sensor_fast() for IRQ handlers lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 11/92] windfarm: decrement client count when unregistering lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 36/92] module: Fix locking in symbol_put_addr() lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 17/92] of/address: Don't loop forever in of_find_matching_node_by_address(). lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 14/92] USB: ftdi_sio: Added custom PID for CustomWare products lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 23/92] IB/uverbs: reject invalid or unknown opcodes lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 27/92] hfs,hfsplus: cache pages correctly between bnode_create and bnode_free lizf@kernel.org - 2016-04-18 13:20 +0200
[PATCH 3.4 05/92] PCI: Add VPD function 0 quirk for Intel Ethernet devices lizf@kernel.org - 2016-04-18 13:30 +0200
Re: [PATCH 3.4 00/92] 3.4.112-rc1 review Guenter Roeck <linux@roeck-us.net> - 2016-04-18 18:40 +0200
Re: [PATCH 3.4 00/92] 3.4.112-rc1 review Zefan Li <lizefan@huawei.com> - 2016-04-19 02:30 +0200
csiph-web