Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1542680 > unrolled thread

Big I/O requests are split into small ones due to unaligned ext4 partition boundary?

Back to article view | Back to linux.kernel

Contents

  Big I/O requests are split into small ones due to unaligned ext4  partition boundary? Dexuan Cui <decui@microsoft.com> - 2016-12-15 13:10 +0100
    Re: Big I/O requests are split into small ones due to unaligned ext4  partition boundary? Ming Lei <tom.leiming@gmail.com> - 2016-12-15 13:50 +0100
      RE: Big I/O requests are split into small ones due to unaligned ext4  partition boundary? Dexuan Cui <decui@microsoft.com> - 2016-12-15 15:10 +0100
        Re: Big I/O requests are split into small ones due to unaligned ext4  partition boundary? Ming Lei <tom.leiming@gmail.com> - 2016-12-16 06:50 +0100

#1542680 — Big I/O requests are split into small ones due to unaligned ext4 partition boundary?

Hi, when I run "mkfs.ext4 /dev/sdc2" in a Linux virtual machine on Hyper-V,
where a disk IOPS=500 limit is applied by me [0],  the command takes much
more time, if the ext4 partition boundary is not properly aligned:

Example 1 [1]: it takes ~7 minutes with average wMB/s = 0.3   (slow)
Example 2 [2]: it takes ~3.5 minutes with average wMB/s = 0.6 (slow)
Example 3 [3]: it takes ~0.5 minute with average wMB/s = 4 (expected)

strace shows the mkfs.ext3 program calls seek()/write() a lot and most of
the writes use 32KB buffers (this should be big enough), and the program
only invokes fsync() once, after it issues all the writes -- the fsync() takes
>99% of the time.

By logging SCSI commands, the SCSI Write(10) command is used here for the
userspace 32KB write:
in example 1, *each* command writes 1 or 2 sectors only (1 sector = 512 bytes);
in example 2, *each* command writes 2 or 4 sectors only;
in example 3, each command writes 1024 sectors.

It looks the kernel block I/O layer can somehow split big user-space buffers
into really small write requests (1, 2, and 4 sectors)?
This looks really strange to me.

Note: in my test, this strange issue happens to 4.4 and the mainline 4.9 kernels,
but the stable 3.18.45 kernel doesn't have the issue, i.e. all the 3 above test
examples can finish in ~0.5 minute.

Any comment?

Thanks!
-- Dexuan


[0] The max IOPS are measured in 8KB increments, meaning the max
 throughput is 8KB * 500 = 4000KB.

[1] This is the partition info of my 20GB disk:
# fdisk -l /dev/sdc
Disk /dev/sdc: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sdc1              1 14281784 14281784  6.8G 82 Linux swap / Solaris
/dev/sdc2       14281785 41929649 27647865 13.2G 83 Linux

Here, start_sector = 14281785, end_sector = 41929649.

[2] start_sector = 14282752, end_sector = 41929649

[3] start_sector = 14282752, end_sector = 41943039

[toc] | [next] | [standalone]

#1542707

On Thu, Dec 15, 2016 at 7:47 PM, Dexuan Cui <decui@microsoft.com> wrote:
> Hi, when I run "mkfs.ext4 /dev/sdc2" in a Linux virtual machine on Hyper-V,
> where a disk IOPS=500 limit is applied by me [0],  the command takes much
> more time, if the ext4 partition boundary is not properly aligned:
>
> Example 1 [1]: it takes ~7 minutes with average wMB/s = 0.3   (slow)
> Example 2 [2]: it takes ~3.5 minutes with average wMB/s = 0.6 (slow)
> Example 3 [3]: it takes ~0.5 minute with average wMB/s = 4 (expected)
>
> strace shows the mkfs.ext3 program calls seek()/write() a lot and most of
> the writes use 32KB buffers (this should be big enough), and the program
> only invokes fsync() once, after it issues all the writes -- the fsync() takes
>>99% of the time.
>
> By logging SCSI commands, the SCSI Write(10) command is used here for the
> userspace 32KB write:
> in example 1, *each* command writes 1 or 2 sectors only (1 sector = 512 bytes);
> in example 2, *each* command writes 2 or 4 sectors only;
> in example 3, each command writes 1024 sectors.
>
> It looks the kernel block I/O layer can somehow split big user-space buffers
> into really small write requests (1, 2, and 4 sectors)?
> This looks really strange to me.
>
> Note: in my test, this strange issue happens to 4.4 and the mainline 4.9 kernels,
> but the stable 3.18.45 kernel doesn't have the issue, i.e. all the 3 above test
> examples can finish in ~0.5 minute.
>
> Any comment?

I remember that we discussed this kind of issue, please see the discussion[1]
and check if the patch[2] can fix your issue.

[1] http://marc.info/?t=145805525500002&r=1&w=2
[2] http://marc.info/?l=linux-kernel&m=145934325429152&w=2


Thanks,
Ming


>
> Thanks!
> -- Dexuan
>
>
> [0] The max IOPS are measured in 8KB increments, meaning the max
>  throughput is 8KB * 500 = 4000KB.
>
> [1] This is the partition info of my 20GB disk:
> # fdisk -l /dev/sdc
> Disk /dev/sdc: 20 GiB, 21474836480 bytes, 41943040 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: dos
> Disk identifier: 0x00000000
>
> Device     Boot    Start      End  Sectors  Size Id Type
> /dev/sdc1              1 14281784 14281784  6.8G 82 Linux swap / Solaris
> /dev/sdc2       14281785 41929649 27647865 13.2G 83 Linux
>
> Here, start_sector = 14281785, end_sector = 41929649.
>
> [2] start_sector = 14282752, end_sector = 41929649
>
> [3] start_sector = 14282752, end_sector = 41943039



-- 
Ming Lei

[toc] | [prev] | [next] | [standalone]

#1542757

> From: Ming Lei [mailto:tom.leiming@gmail.com]
> Sent: Thursday, December 15, 2016 20:43
> 
> On Thu, Dec 15, 2016 at 7:47 PM, Dexuan Cui <decui@microsoft.com> wrote:
> > Hi, when I run "mkfs.ext4 /dev/sdc2" in a Linux virtual machine on Hyper-V,
> > where a disk IOPS=500 limit is applied by me [0],  the command takes much
> > more time, if the ext4 partition boundary is not properly aligned:
> >
> > Example 1 [1]: it takes ~7 minutes with average wMB/s = 0.3   (slow)
> > Example 2 [2]: it takes ~3.5 minutes with average wMB/s = 0.6 (slow)
> > Example 3 [3]: it takes ~0.5 minute with average wMB/s = 4 (expected)
> >
> > strace shows the mkfs.ext3 program calls seek()/write() a lot and most of
> > the writes use 32KB buffers (this should be big enough), and the program
> > only invokes fsync() once, after it issues all the writes -- the fsync() takes
> >>99% of the time.
> >
> > By logging SCSI commands, the SCSI Write(10) command is used here for the
> > userspace 32KB write:
> > in example 1, *each* command writes 1 or 2 sectors only (1 sector = 512
> bytes);
> > in example 2, *each* command writes 2 or 4 sectors only;
> > in example 3, each command writes 1024 sectors.
> >
> > It looks the kernel block I/O layer can somehow split big user-space buffers
> > into really small write requests (1, 2, and 4 sectors)?
> > This looks really strange to me.
> >
> > Note: in my test, this strange issue happens to 4.4 and the mainline 4.9 kernels,
> > but the stable 3.18.45 kernel doesn't have the issue, i.e. all the 3 above test
> > examples can finish in ~0.5 minute.
> >
> > Any comment?
> 
> I remember that we discussed this kind of issue, please see the discussion[1]
> and check if the patch[2] can fix your issue.
> 
> [1] http://marc.info/?t=145805525500002&r=1&w=2
> [2] http://marc.info/?l=linux-kernel&m=145934325429152&w=2
> 
> Ming
 
Thank you very much, Ming! The patch can fix my issue!
It looks your patch was not merged into the upstream somehow.
Would you please submit the patch again?

Thanks,
-- Dexuan

[toc] | [prev] | [next] | [standalone]

#1543207

On Thu, Dec 15, 2016 at 9:53 PM, Dexuan Cui <decui@microsoft.com> wrote:
>> From: Ming Lei [mailto:tom.leiming@gmail.com]
>> Sent: Thursday, December 15, 2016 20:43
>>
>> On Thu, Dec 15, 2016 at 7:47 PM, Dexuan Cui <decui@microsoft.com> wrote:
>> > Hi, when I run "mkfs.ext4 /dev/sdc2" in a Linux virtual machine on Hyper-V,
>> > where a disk IOPS=500 limit is applied by me [0],  the command takes much
>> > more time, if the ext4 partition boundary is not properly aligned:
>> >
>> > Example 1 [1]: it takes ~7 minutes with average wMB/s = 0.3   (slow)
>> > Example 2 [2]: it takes ~3.5 minutes with average wMB/s = 0.6 (slow)
>> > Example 3 [3]: it takes ~0.5 minute with average wMB/s = 4 (expected)
>> >
>> > strace shows the mkfs.ext3 program calls seek()/write() a lot and most of
>> > the writes use 32KB buffers (this should be big enough), and the program
>> > only invokes fsync() once, after it issues all the writes -- the fsync() takes
>> >>99% of the time.
>> >
>> > By logging SCSI commands, the SCSI Write(10) command is used here for the
>> > userspace 32KB write:
>> > in example 1, *each* command writes 1 or 2 sectors only (1 sector = 512
>> bytes);
>> > in example 2, *each* command writes 2 or 4 sectors only;
>> > in example 3, each command writes 1024 sectors.
>> >
>> > It looks the kernel block I/O layer can somehow split big user-space buffers
>> > into really small write requests (1, 2, and 4 sectors)?
>> > This looks really strange to me.
>> >
>> > Note: in my test, this strange issue happens to 4.4 and the mainline 4.9 kernels,
>> > but the stable 3.18.45 kernel doesn't have the issue, i.e. all the 3 above test
>> > examples can finish in ~0.5 minute.
>> >
>> > Any comment?
>>
>> I remember that we discussed this kind of issue, please see the discussion[1]
>> and check if the patch[2] can fix your issue.
>>
>> [1] http://marc.info/?t=145805525500002&r=1&w=2
>> [2] http://marc.info/?l=linux-kernel&m=145934325429152&w=2
>>
>> Ming
>
> Thank you very much, Ming! The patch can fix my issue!
> It looks your patch was not merged into the upstream somehow.
> Would you please submit the patch again?

Yeah, will do, and thanks for your test!



Thanks,
Ming Lei

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web