Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #52508

Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes

Cross-posted to 2 groups.

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

Control: tags -1 + fixed-upstream patch jessie
Control: found -1 3.16.7-ckt20-1+deb8u3
Control: fixed -1 4.1.1-1~exp1

Hi,

so I've found out that this is actually a bug in LIO (the target) not
the initiator.

Problem is: LIO in Linux up to 4.0 uses vfs_writev to write out blocks
when using the fileio backend, so this has a limit of 1024 iovecs (as
per UIO_MAXIOV in include/uapi/linux/uio.h), each at most a page in
size, so that gives us a maximum of 4 MiB in data that can be processed
per request. (At 4 kiB page size.)

Older versions of the Linux software initiator had a hard limit of
512 kiB per request, which means that at most 128 iovec entries were
used, which fits perfectly. Newer versions of the Linux iSCSI initiator
don't have this hard-coded limit but rather use the value supplied by
the target. (This is correct behavior by the initiator, so there's no
bug there, against what I initially assumed.)

Problem now is that LIO with the fileio backend assumes that 8 MiB may
be transfered at the same time, because (according to comments in
drivers/target/target_core_file.h) they assume for some reason
unbeknownst to me that the maximum number of iovecs is 2048.

Note that this also means that any non-Linux initiator that properly
follows the target-supplied values for the maximum I/O size will run
into the same problem and cause I/O errors, even if it behaves
properly.

This problem doesn't affect upstream anymore, because they have
rewritten LIO to use vfs_iter_write, which doesn't have such
limitations, but was only introduced after 3.16. Backporting this would
be too much, and probably ABI-incompatible.

Fortunately, there's a much easier way to fix this, by just lowering
the limit in drivers/target/target_core_file.h to 4 MiB. I've tested
that and the limit will be properly set by LIO and newer initiators
won't choke on that, so that fixes the bug. See the attached patch.

CAVEAT: there is a slight problem with this change, and I don't know
what the best solution here is: the optimal_sectors setting for fileio
disks on people's existing setups is likely to be 16384, because that
corresponds to 8 MiB (the previous max value, which is the default for
optimal_sectors if no other value is set) - but that will cause the
kernel to refuse that setting, because it's now larger than the maximum
allowed value. If you use targetcli 3 (not part of Jessie, but you can
install e.g. the version from sid) then that will fail to set up the
target properly, because it will abort as soon as it notices that it
can't make the setting. (Leftover targetcli 2 from Wheezy upgrades
should not be affected as badly as far as I can tell, because the
startup scripts seem to ignore errors. But I haven't tested that.) So
that leaves the situation that without this fix, 3.16 kernels produce
I/O errors when used with initiators that respect the kernel's setting,
but with the fix the target configuration needs to be updated. (Of
course, one could also patch the kernel to ignore the specific value of
16384 for optimal_sectors if fileio is used as a backend and print a
warning.) Don't know what you'd prefer here.

Also note that this likely also affects the kernel in Wheezy, although
I haven't done any tests in that direction.

Regards,
Christian

PS, for reference, upstream discussion on the initiator mailing list
that resulted in me finding out that it's not the initiator but the
target that was the problem:
https://groups.google.com/forum/#!topic/open-iscsi/UE2JJfDmQ7w

Back to linux.debian.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes Christian Seiler <christian@iwakd.de> - 2016-01-27 23:40 +0100
  Bug#805252: linux-image-4.2.0-1-amd64: I/O errors when writing to iSCSI volumes Christian Seiler <christian@iwakd.de> - 2016-01-30 15:40 +0100

csiph-web