Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #52481 > unrolled thread

Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes

Back to article view | Back to linux.debian.kernel

Contents

  Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes Christian Seiler <christian@iwakd.de> - 2016-01-27 23:40 +0100
    Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes Christian Seiler <christian@iwakd.de> - 2016-01-30 15:40 +0100

#52481 — Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes

[Multipart message — attachments visible in raw view] — view raw

Control: tags -1 - fixed-upstream patch
Control: notfixed -1 4.4~rc5-1~exp1

Hi,

I'm sorry, but the commit in question doesn't help. I just got around
to testing this with 4.4-1~exp1, and I verified that the sources do
indeed contain the aforementioned patch, and the problem persists. I
also tried this with the most recent upstream kernel git tree [1] and
the problem also persists there.

Using

   dd if=/dev/zero of=test.dat

it will reproducibly cause lots of I/O errors *before* the disk runs
full.

As already said in the original report, the most recent 3.16 kernel
that comes with Jessie does not show this problem, even when used
with exactly the same userland (up to date sid), there the dd
command will just create a large file until the disk is full (as
expected).

I'll ask open-iscsi upstream for some help with this, but wanted to
make sure this is properly tracked in Debian's bugtracker.

Regards,
Christian

[1] Latest commit at time of testing:
    03c21cb775a313f1ff19be59c5d02df3e3526471

    Built the custom kernel via make-kpkg, using 4.4-1~exp1's
    config as a basis and then running make oldconfig.

[toc] | [next] | [standalone]

#52508

[Multipart message — attachments visible in raw view] — view raw

Control: tags -1 + fixed-upstream patch jessie
Control: found -1 3.16.7-ckt20-1+deb8u3
Control: fixed -1 4.1.1-1~exp1

Hi,

so I've found out that this is actually a bug in LIO (the target) not
the initiator.

Problem is: LIO in Linux up to 4.0 uses vfs_writev to write out blocks
when using the fileio backend, so this has a limit of 1024 iovecs (as
per UIO_MAXIOV in include/uapi/linux/uio.h), each at most a page in
size, so that gives us a maximum of 4 MiB in data that can be processed
per request. (At 4 kiB page size.)

Older versions of the Linux software initiator had a hard limit of
512 kiB per request, which means that at most 128 iovec entries were
used, which fits perfectly. Newer versions of the Linux iSCSI initiator
don't have this hard-coded limit but rather use the value supplied by
the target. (This is correct behavior by the initiator, so there's no
bug there, against what I initially assumed.)

Problem now is that LIO with the fileio backend assumes that 8 MiB may
be transfered at the same time, because (according to comments in
drivers/target/target_core_file.h) they assume for some reason
unbeknownst to me that the maximum number of iovecs is 2048.

Note that this also means that any non-Linux initiator that properly
follows the target-supplied values for the maximum I/O size will run
into the same problem and cause I/O errors, even if it behaves
properly.

This problem doesn't affect upstream anymore, because they have
rewritten LIO to use vfs_iter_write, which doesn't have such
limitations, but was only introduced after 3.16. Backporting this would
be too much, and probably ABI-incompatible.

Fortunately, there's a much easier way to fix this, by just lowering
the limit in drivers/target/target_core_file.h to 4 MiB. I've tested
that and the limit will be properly set by LIO and newer initiators
won't choke on that, so that fixes the bug. See the attached patch.

CAVEAT: there is a slight problem with this change, and I don't know
what the best solution here is: the optimal_sectors setting for fileio
disks on people's existing setups is likely to be 16384, because that
corresponds to 8 MiB (the previous max value, which is the default for
optimal_sectors if no other value is set) - but that will cause the
kernel to refuse that setting, because it's now larger than the maximum
allowed value. If you use targetcli 3 (not part of Jessie, but you can
install e.g. the version from sid) then that will fail to set up the
target properly, because it will abort as soon as it notices that it
can't make the setting. (Leftover targetcli 2 from Wheezy upgrades
should not be affected as badly as far as I can tell, because the
startup scripts seem to ignore errors. But I haven't tested that.) So
that leaves the situation that without this fix, 3.16 kernels produce
I/O errors when used with initiators that respect the kernel's setting,
but with the fix the target configuration needs to be updated. (Of
course, one could also patch the kernel to ignore the specific value of
16384 for optimal_sectors if fileio is used as a backend and print a
warning.) Don't know what you'd prefer here.

Also note that this likely also affects the kernel in Wheezy, although
I haven't done any tests in that direction.

Regards,
Christian

PS, for reference, upstream discussion on the initiator mailing list
that resulted in me finding out that it's not the initiator but the
target that was the problem:
https://groups.google.com/forum/#!topic/open-iscsi/UE2JJfDmQ7w

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web