Path: csiph.com!3.eu.feeder.erje.net!feeder.erje.net!news.unit0.net!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod From: Juergen Gross Newsgroups: linux.debian.bugs.dist,linux.debian.kernel Subject: Bug#850425: mpt3sas "swiotlb buffer is full" problem only under Xen Date: Mon, 11 Mar 2019 15:00:02 +0100 Message-ID: References: X-Mailbox-Line: From debian-bugs-dist-request@lists.debian.org Mon Mar 11 13:51:08 2019 Old-Return-Path: X-Spam-Flag: NO X-Spam-Score: -4.199 Reply-To: Juergen Gross , 850425@bugs.debian.org Resent-To: debian-bugs-dist@lists.debian.org Resent-Cc: Debian Kernel Team X-Debian-Pr-Message: followup 850425 X-Debian-Pr-Package: src:linux X-Debian-Pr-Keywords: upstream patch moreinfo X-Debian-Pr-Source: linux X-Spam-Bayes: score:0.0000 Tokens: new, 182; hammy, 150; neutral, 232; spammy, 0. spammytokens: hammytokens:0.000-+--HOpenpgp:preference, 0.000-+--HOpenpgp:signencrypt, 0.000-+--HAutocrypt:keydata, 0.000-+--HAutocrypt:addr, 0.000-+--HAutocrypt:prefer-encrypt Openpgp: preference=signencrypt Autocrypt: addr=jgross@suse.com; prefer-encrypt=mutual; keydata= mQENBFOMcBYBCACgGjqjoGvbEouQZw/ToiBg9W98AlM2QHV+iNHsEs7kxWhKMjrioyspZKOB ycWxw3ie3j9uvg9EOB3aN4xiTv4qbnGiTr3oJhkB1gsb6ToJQZ8uxGq2kaV2KL9650I1SJve dYm8Of8Zd621lSmoKOwlNClALZNew72NjJLEzTalU1OdT7/i1TXkH09XSSI8mEQ/ouNcMvIJ NwQpd369y9bfIhWUiVXEK7MlRgUG6MvIj6Y3Am/BBLUVbDa4+gmzDC9ezlZkTZG2t14zWPvx XP3FAp2pkW0xqG7/377qptDmrk42GlSKN4z76ELnLxussxc7I2hx18NUcbP8+uty4bMxABEB AAG0H0p1ZXJnZW4gR3Jvc3MgPGpncm9zc0BzdXNlLmNvbT6JATkEEwECACMFAlOMcK8CGwMH CwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRCw3p3WKL8TL8eZB/9G0juS/kDY9LhEXseh mE9U+iA1VsLhgDqVbsOtZ/S14LRFHczNd/Lqkn7souCSoyWsBs3/wO+OjPvxf7m+Ef+sMtr0 G5lCWEWa9wa0IXx5HRPW/ScL+e4AVUbL7rurYMfwCzco+7TfjhMEOkC+va5gzi1KrErgNRHH kg3PhlnRY0Udyqx++UYkAsN4TQuEhNN32MvN0Np3WlBJOgKcuXpIElmMM5f1BBzJSKBkW0Jc Wy3h2Wy912vHKpPV/Xv7ZwVJ27v7KcuZcErtptDevAljxJtE7aJG6WiBzm+v9EswyWxwMCIO RoVBYuiocc51872tRGywc03xaQydB+9R7BHPuQENBFOMcBYBCADLMfoA44MwGOB9YT1V4KCy vAfd7E0BTfaAurbG+Olacciz3yd09QOmejFZC6AnoykydyvTFLAWYcSCdISMr88COmmCbJzn sHAogjexXiif6ANUUlHpjxlHCCcELmZUzomNDnEOTxZFeWMTFF9Rf2k2F0Tl4E5kmsNGgtSa aMO0rNZoOEiD/7UfPP3dfh8JCQ1VtUUsQtT1sxos8Eb/HmriJhnaTZ7Hp3jtgTVkV0ybpgFg w6WMaRkrBh17mV0z2ajjmabB7SJxcouSkR0hcpNl4oM74d2/VqoW4BxxxOD1FcNCObCELfIS auZx+XT6s+CE7Qi/c44ibBMR7hyjdzWbABEBAAGJAR8EGAECAAkFAlOMcBYCGwwACgkQsN6d 1ii/Ey9D+Af/WFr3q+bg/8v5tCknCtn92d5lyYTBNt7xgWzDZX8G6/pngzKyWfedArllp0Pn fgIXtMNV+3t8Li1Tg843EXkP7+2+CQ98MB8XvvPLYAfW8nNDV85TyVgWlldNcgdv7nn1Sq8g HwB2BHdIAkYce3hEoDQXt/mKlgEGsLpzJcnLKimtPXQQy9TxUaLBe9PInPd+Ohix0XOlY+Uk QFEx50Ki3rSDl2Zt2tnkNYKUCvTJq7jvOlaPd6d/W0tZqpyy7KVay+K4aMobDsodB3dvEAs6 ScCnh03dDAFgIq5nsB11j3KPKdVoPlfucX2c7kGNH+LUMbzqV6beIENfNexkOfxHf4kBrQQY AQgAIBYhBIUSZ3Lo9gSUpdCX97DendYovxMvBQJa3fDQAhsCAIEJELDendYovxMvdiAEGRYI AB0WIQRTLbB6QfY48x44uB6AXGG7T9hjvgUCWt3w0AAKCRCAXGG7T9hjvk2LAP99B/9FenK/ 1lfifxQmsoOrjbZtzCS6OKxPqOLHaY47BgEAqKKn36YAPpbk09d2GTVetoQJwiylx/Z9/mQI CUbQMg1pNQf9EjA1bNcMbnzJCgt0P9Q9wWCLwZa01SnQWFz8Z4HEaKldie+5bHBL5CzVBrLv 81tqX+/j95llpazzCXZW2sdNL3r8gXqrajSox7LR2rYDGdltAhQuISd2BHrbkQVEWD4hs7iV 1KQHe2uwXbKlguKPhk5ubZxqwsg/uIHw0qZDk+d0vxjTtO2JD5Jv/CeDgaBX4Emgp0NYs8IC UIyKXBtnzwiNv4cX9qKlz2Gyq9b+GdcLYZqMlIBjdCz0yJvgeb3WPNsCOanvbjelDhskx9gd 6YUUFFqgsLtrKpCNyy203a58g2WosU9k9H+LcheS37Ph2vMVTISMszW9W8gyORSgmw== User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------23C79CE5AC5067CA1897EDA8" Content-Language: de-DE X-Debian-Message: from BTS X-Mailing-List: archive/latest/1526494 List-ID: List-URL: Approved: robomod@news.nic.it Lines: 130 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Cc: Andrew Cooper , 850425@bugs.debian.org, Andy Smith , Boris Ostrovsky , Konrad Rzeszutek Wilk X-Original-Date: Mon, 11 Mar 2019 14:50:13 +0100 X-Original-Message-ID: <3644ce89-7eb6-e505-6732-3891d27d8128@suse.com> X-Original-References: <0f8d4f04-6075-5c31-6b97-c59a5d8589a7@knorrie.org> <7261253e-5dc1-60c9-7b64-f592aa9fd787@citrix.com> <20170106113711.GR1804@bitfolk.com> <7261253e-5dc1-60c9-7b64-f592aa9fd787@citrix.com> Xref: csiph.com linux.debian.bugs.dist:949435 linux.debian.kernel:63554 This is a multi-part message in MIME format. --------------23C79CE5AC5067CA1897EDA8 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit On 10/03/2019 23:03, Andrew Cooper wrote: > On 10/03/2019 21:35, Hans van Kranenburg wrote: >> found -1 4.19.20-1 >> thanks >> >> Hi, >> >> Reviving a thing from Jan 2017 here. I don't have this thread in my >> mailbox, so no inline quotes. >> >> I just installed some HP z820 workstation and rebooted it into Xen >> 4.11.1+26-g87f51bf366-3 with linux 4.19.20-1 as dom0 kernel. >> >> During boot I'm greeted by a long list of... >> >> [ 14.518793] mpt3sas 0000:02:00.0: swiotlb buffer is full (sz: 65536 >> bytes) >> [ 14.518899] mpt3sas 0000:02:00.0: swiotlb buffer is full >> [ 14.518956] mpt3sas 0000:02:00.0: swiotlb buffer is full (sz: 65536 >> bytes) >> [ 14.518988] sd 6:0:3:0: pci_map_sg failed: request for 786432 bytes! >> [ 14.519081] mpt3sas 0000:02:00.0: swiotlb buffer is full >> [ 14.519309] sd 6:0:1:0: pci_map_sg failed: request for 1310720 bytes! >> [ 14.524611] mpt3sas 0000:02:00.0: swiotlb buffer is full (sz: 65536 >> bytes) >> [ 14.527309] mpt3sas 0000:02:00.0: swiotlb buffer is full >> [ 14.527405] sd 6:0:3:0: pci_map_sg failed: request for 786432 bytes! >> [...] >> >> ...and some hangs here and there. This indeed did not happen when >> booting just Linux, without Xen. >> >> Some searching brought me to this Debian bug. So, thanks for writing >> down all kinds of research here already. Even if it's not fixed upstream >> yet, this helps a lot. :-) >> >> Using dom0_mem=2GiB,max:4GiB instead of dom0_mem=2GiB,max:2GiB (which I >> started with) makes the errors go away, so workaround confirmed. >> >> I can try any of the linked patches, but I see that in message 54, >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850425#54 >> Andrew says: "IIRC, they were essentially rejected,". Next message, Ian >> asks "Do you have a reference ?", but I don't see any fup on that. >> >> I think I'm fine with this workaround. >> >> If someone will ever work on the upstream patches, then this is just to >> let know that I might be able to help testing. However, I only have one >> of this type of box and it's gonna be installed as server at some >> non-profit organization without OOB access, replacing even older donated >> hardware, so, it will be kinda limited... :) > > I think > https://lists.xen.org/archives/html/xen-devel/2014-12/msg00699.html is > the last attempt David made to upstream the fixes. Attached is a rebase of the last part missing. Should apply on top of 5.0 kernel, 4.20 should be okay, too. Earlier kernels will miss some prerequisites. Juergen --------------23C79CE5AC5067CA1897EDA8 Content-Type: text/x-patch; name="swiotlb.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="swiotlb.patch" From: David Vrabel Date: Mon, 11 Mar 2019 14:40:00 +0100 Subject: [PATCH] x86/xen: assume a 64-bit DMA mask is required On a Xen PV guest the DMA addresses and physical addresses are not 1:1 (such as Xen PV guests) and the generic dma_get_required_mask() does not return the correct mask (since it uses max_pfn). Some device drivers (such as mptsas, mpt2sas) use dma_get_required_mask() to set the device's DMA mask to allow them to use only 32-bit DMA addresses in hardware structures. This results in unnecessary use of the SWIOTLB if DMA addresses are more than 32-bits, impacting performance significantly. We could base the DMA mask on the maximum MFN but: a) The hypercall op to get the maximum MFN (XENMEM_maximum_ram_page) will truncate the result to an int in 32-bit guests. b) Future uses of the IOMMU in Xen may map frames at bus addresses above the end of RAM. So, just assume a 64-bit DMA mask is always required. Signed-off-by: David Vrabel Reviewed-by: Juergen Gross --- drivers/xen/swiotlb-xen.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index bb7888429be6..75e6e440d982 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -680,6 +680,11 @@ xen_swiotlb_get_sgtable(struct device *dev, struct sg_table *sgt, return dma_common_get_sgtable(dev, sgt, cpu_addr, handle, size, attrs); } +static u64 xen_swiotlb_get_required_mask(struct device *dev) +{ + return DMA_BIT_MASK(64); +} + const struct dma_map_ops xen_swiotlb_dma_ops = { .alloc = xen_swiotlb_alloc_coherent, .free = xen_swiotlb_free_coherent, @@ -694,4 +699,5 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = xen_swiotlb_dma_mmap, .get_sgtable = xen_swiotlb_get_sgtable, + .get_required_mask = xen_swiotlb_get_required_mask, }; --------------23C79CE5AC5067CA1897EDA8--