Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #225554 > unrolled thread

VMs on external storage CPU overloading

Back to article view | Back to linux.debian.user

Contents

  VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-07-30 16:20 +0200
    Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-07-30 16:50 +0200
      Re: VMs on external storage CPU overloading Dan Ritter <dsr@randomstring.org> - 2020-07-30 17:30 +0200
        Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-07-30 17:50 +0200
        Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-08-03 22:30 +0200
          Re: VMs on external storage CPU overloading Michael Stone <mstone@debian.org> - 2020-08-03 23:10 +0200
            Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-08-04 08:50 +0200
          Re: VMs on external storage CPU overloading David Christensen <dpchrist@holgerdanske.com> - 2020-08-03 23:30 +0200
    Re: VMs on external storage CPU overloading Dan Ritter <dsr@randomstring.org> - 2020-07-30 16:50 +0200
    Re: VMs on external storage CPU overloading deloptes <deloptes@gmail.com> - 2020-07-30 22:10 +0200
    Re: VMs on external storage CPU overloading Stefan Monnier <monnier@iro.umontreal.ca> - 2020-07-30 23:00 +0200
      Re: VMs on external storage CPU overloading deloptes <deloptes@gmail.com> - 2020-07-31 08:00 +0200
        Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-07-31 10:40 +0200
          Re: VMs on external storage CPU overloading Andrei POPESCU <andreimpopescu@gmail.com> - 2020-08-02 09:40 +0200
        Re: VMs on external storage CPU overloading Stefan Monnier <monnier@iro.umontreal.ca> - 2020-07-31 15:00 +0200
          Re: VMs on external storage CPU overloading deloptes <deloptes@gmail.com> - 2020-08-01 01:30 +0200
            Re: VMs on external storage CPU overloading Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 06:00 +0200
          Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-08-01 10:50 +0200
            Re: VMs on external storage CPU overloading Andrew Cater <amacater@gmail.com> - 2020-08-01 11:50 +0200
              Re: VMs on external storage CPU overloading john doe <johndoe65534@mail.com> - 2020-08-01 12:10 +0200
            Re: VMs on external storage CPU overloading Stefan Monnier <monnier@iro.umontreal.ca> - 2020-08-03 06:00 +0200

Page 1 of 2  [1] 2  Next page →

#225554 — VMs on external storage CPU overloading

Hi Debians,

I have limited space on my box, so I would like to install VMs onto an
external storage (usbkey 3.0) using Libvirt.

In the below command '/mnt/usbkey01' is the mountpoint for my usbkey:

virt-install --name=try01 --ram=1024
--disk=path=/mnt/usbkey01/machines/try01,size=6


My CPU overloads like crazy.

The overloading is related by the use of this usbkey but I don't
understand why?:
- Is this a Debian issue
- Is this a Libvirt issue
- Is this expected because I'm using virtualisation with KVM enabled


Other then increasing the size of my internal storage, can I do
something about it?


If I change the path to '/var/lib/machines/try01' (internal storage) it
works flowlessly and my CPU does not overload.


Any input is appriciated.

--
John Doe

[toc] | [next] | [standalone]

#225556

On 7/30/2020 4:41 PM, Dan Ritter wrote:
> john doe wrote:
>> Hi Debians,
>>
>> I have limited space on my box, so I would like to install VMs onto an
>> external storage (usbkey 3.0) using Libvirt.
>>
>> In the below command '/mnt/usbkey01' is the mountpoint for my usbkey:
>>
>> virt-install --name=try01 --ram=1024
>> --disk=path=/mnt/usbkey01/machines/try01,size=6
>>
>>
>> My CPU overloads like crazy.
>>
>> The overloading is related by the use of this usbkey but I don't
>> understand why?:
>> - Is this a Debian issue
>> - Is this a Libvirt issue
>> - Is this expected because I'm using virtualisation with KVM enabled
>>
>>
>> Other then increasing the size of my internal storage, can I do
>> something about it?
>>
>>
>> If I change the path to '/var/lib/machines/try01' (internal storage) it
>> works flowlessly and my CPU does not overload.
>
> Two things:
>
> - your CPU load, depending on how you measure it, may include
>    processes which are waiting for I/O. This I/O is very slow,
>    so.
>
> - not all USB3 devices are the same. Many implement a
>    single-request system, where the kernel is only allowed to
>    make a read or write call, then wait for the response, then
>    issue the next call. Anything describing itself as a "key"
>    is particularly likely to have this.
>

Is there any way to verify this (command wize)?

> Can you change your external storage?
>

Any suggestion?


Thanks DSR.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#225562

john doe wrote: 
> On 7/30/2020 4:41 PM, Dan Ritter wrote:
> > Two things:
> > 
> > - your CPU load, depending on how you measure it, may include
> >    processes which are waiting for I/O. This I/O is very slow,
> >    so.
> > 
> > - not all USB3 devices are the same. Many implement a
> >    single-request system, where the kernel is only allowed to
> >    make a read or write call, then wait for the response, then
> >    issue the next call. Anything describing itself as a "key"
> >    is particularly likely to have this.
> > 
> 
> Is there any way to verify this (command wize)?

$ usb-devices

will show

 Driver=usb-storage

for a slow device, and

 Driver=uas

for a fast device.

> > Can you change your external storage?
> 
> Any suggestion?

You're looking for a device that has UASP support rather than
just USB Mass Storage. Some vendors make it easier to search
for this than others, e.g.

https://www.startech.com/HDD/Enclosures/?filter_UASP_YN=Yes

will show you all the StarTech USB3 enclosures that do UASP.

I don't have any specific recommendations.

-dsr-

[toc] | [prev] | [next] | [standalone]

#225563

On 7/30/2020 5:19 PM, Dan Ritter wrote:
> john doe wrote:
>> On 7/30/2020 4:41 PM, Dan Ritter wrote:
>>> Two things:
>>>
>>> - your CPU load, depending on how you measure it, may include
>>>     processes which are waiting for I/O. This I/O is very slow,
>>>     so.
>>>
>>> - not all USB3 devices are the same. Many implement a
>>>     single-request system, where the kernel is only allowed to
>>>     make a read or write call, then wait for the response, then
>>>     issue the next call. Anything describing itself as a "key"
>>>     is particularly likely to have this.
>>>
>>
>> Is there any way to verify this (command wize)?
>
> $ usb-devices
>
> will show
>
>   Driver=usb-storage
>

Yep, I've got this one, a slow device.

> for a slow device, and
>
>   Driver=uas
>
> for a fast device.
>
>>> Can you change your external storage?
>>
>> Any suggestion?
>
> You're looking for a device that has UASP support rather than
> just USB Mass Storage. Some vendors make it easier to search
> for this than others, e.g.
>
> https://www.startech.com/HDD/Enclosures/?filter_UASP_YN=Yes
>
> will show you all the StarTech USB3 enclosures that do UASP.
>
> I don't have any specific recommendations.
>

Awsome, thanks again dsr.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#225787

On 7/30/2020 5:19 PM, Dan Ritter wrote:
> john doe wrote:
>> On 7/30/2020 4:41 PM, Dan Ritter wrote:
>>> Two things:
>>>
>>> - your CPU load, depending on how you measure it, may include
>>>     processes which are waiting for I/O. This I/O is very slow,
>>>     so.
>>>
>>> - not all USB3 devices are the same. Many implement a
>>>     single-request system, where the kernel is only allowed to
>>>     make a read or write call, then wait for the response, then
>>>     issue the next call. Anything describing itself as a "key"
>>>     is particularly likely to have this.
>>>
>>
>> Is there any way to verify this (command wize)?
>
> $ usb-devices
>
> will show
>
>   Driver=usb-storage
>
> for a slow device, and
>
>   Driver=uas
>
> for a fast device.
>
>>> Can you change your external storage?
>>
>> Any suggestion?
>
> You're looking for a device that has UASP support rather than
> just USB Mass Storage. Some vendors make it easier to search
> for this than others, e.g.
>
> https://www.startech.com/HDD/Enclosures/?filter_UASP_YN=Yes
>
> will show you all the StarTech USB3 enclosures that do UASP.
>
> I don't have any specific recommendations.
>

I'm settleing on buying (1), Google says that it is UASP compatible.


The URL (1) is not stating explicetly that it is UASP compatible, how
can I infer that it is realy the case?


Thanks to all for the feedback.

1)
https://www.corsair.com/eu/en/Categories/Products/Storage/USB-Drives/flash-voyager-gtx-3-1-config/p/CMFVYGTX3C-128GB#tab-tech-specs

--
John Doe

[toc] | [prev] | [next] | [standalone]

#225789

On Mon, Aug 03, 2020 at 10:21:01PM +0200, john doe wrote:
>I'm settleing on buying (1), Google says that it is UASP compatible.

Is there some reason you're set on buying a thumb drive form factor?

[toc] | [prev] | [next] | [standalone]

#225808

On 8/3/2020 11:00 PM, Michael Stone wrote:
> On Mon, Aug 03, 2020 at 10:21:01PM +0200, john doe wrote:
>> I'm settleing on buying (1), Google says that it is UASP compatible.
>
> Is there some reason you're set on buying a thumb drive form factor?
>

Mainly the form factor and the ease of use.


I guess you are hinting out at something like (1,2) (Sata3 SSD 2.5 along
with a usb 3.0 to sata3 adapter (UASP compatible))?


This approach would be more flexible as I could only change the drive
without having to buy a new thumdrive if my capacity requirements are
increasing.
Looking more into this, in my case this could also have the benefit of
being able to share that drive across those single boards by using a usb
3.0 hub (UASP compatible).

It is worth looking at, thank you.


1)
https://www.amazon.com/StarTech-com-10Gbps-Adapter-Cable-Drives/dp/B00XLAZODE/ref=sr_1_4?dchild=1&keywords=uasp&qid=1596522233&s=computers-intl-ship&sr=1-4
2)
https://www.amazon.com/Kingston-120GB-Solid-SA400S37-120G/dp/B01N6JQS8C/ref=sr_1_1?dchild=1&keywords=ssd+128&qid=1596522367&s=electronics&sr=1-1


--
John Doe

[toc] | [prev] | [next] | [standalone]

#225790

On 2020-07-30 07:14, john doe wrote:

 > I have limited space on my box, so I would like to install VMs onto an
 > external storage (usbkey 3.0) using Libvirt.


On 2020-07-31 01:32, john doe wrote:
 > I have a single board with only USB 3.0


What is the make and model of your computer?  Processor?  Memory 
module(s)?  Storage device(s)?


What Debian version?  System information (uname -a)?


How do you use the computer?


What use-case are you trying to add or improve?


On 2020-08-03 13:21, john doe wrote:

> I'm settleing on buying (1), Google says that it is UASP compatible.
> 
> 
> The URL (1) is not stating explicetly that it is UASP compatible, how
> can I infer that it is realy the case?
> 
> 
> Thanks to all for the feedback.
> 
> 1)
> https://www.corsair.com/eu/en/Categories/Products/Storage/USB-Drives/flash-voyager-gtx-3-1-config/p/CMFVYGTX3C-128GB#tab-tech-specs 


Contact Corsair technical support and ask them to post UASP support, or 
lack thereof, on their USB product specification pages.


David

[toc] | [prev] | [next] | [standalone]

#225558

john doe wrote: 
> Hi Debians,
> 
> I have limited space on my box, so I would like to install VMs onto an
> external storage (usbkey 3.0) using Libvirt.
> 
> In the below command '/mnt/usbkey01' is the mountpoint for my usbkey:
> 
> virt-install --name=try01 --ram=1024
> --disk=path=/mnt/usbkey01/machines/try01,size=6
> 
> 
> My CPU overloads like crazy.
> 
> The overloading is related by the use of this usbkey but I don't
> understand why?:
> - Is this a Debian issue
> - Is this a Libvirt issue
> - Is this expected because I'm using virtualisation with KVM enabled
> 
> 
> Other then increasing the size of my internal storage, can I do
> something about it?
> 
> 
> If I change the path to '/var/lib/machines/try01' (internal storage) it
> works flowlessly and my CPU does not overload.

Two things:

- your CPU load, depending on how you measure it, may include
  processes which are waiting for I/O. This I/O is very slow,
  so.

- not all USB3 devices are the same. Many implement a
  single-request system, where the kernel is only allowed to
  make a read or write call, then wait for the response, then
  issue the next call. Anything describing itself as a "key"
  is particularly likely to have this.

Can you change your external storage?

-dsr-

[toc] | [prev] | [next] | [standalone]

#225572

john doe wrote:

> Other then increasing the size of my internal storage, can I do
> something about it?

Some boards have SATA or eSATA port that would match external USB-drive with
SATA/eSATA. My one did not have it on board, but I added a 2 Port card with
SATA and connect a rather old usb2 box with SATA. It beats even USB3.0,
which I am not sure I have on this PC. Of course it depends also on the
disk and controller in between, but I prefer the external SATA connection
I don't know if it's possible to do on your end.

regards

[toc] | [prev] | [next] | [standalone]

#225576

> My CPU overloads like crazy.

What makes you think so?  Does it overheat?  Is the "load average" too high?
Where is the CPU time spent (i.e. e.g. what does `top` say, is it mostly
in "wait"?  "kernel"?  "user"?)?

Have you tried to install a similar (tho probably smaller) VM on your
internal storage just in order to see if the problem comes from the
external storage or from the VM itself?


        Stefan

[toc] | [prev] | [next] | [standalone]

#225586

Stefan Monnier wrote:

> What makes you think so?  Does it overheat?  Is the "load average" too
> high? Where is the CPU time spent (i.e. e.g. what does `top` say, is it
> mostly in "wait"?  "kernel"?  "user"?)?
> 
> Have you tried to install a similar (tho probably smaller) VM on your
> internal storage just in order to see if the problem comes from the
> external storage or from the VM itself?

IMO, allthose question as irrelevant - the problem is the IO. You can try it
easily. get USB 2.0 and put a VM on it - try to start it - you will know
what OP means. Obviously also USB3 are not always USB3 :D

[toc] | [prev] | [next] | [standalone]

#225591

Answering here to Stefanand Deloptes.

On 7/31/2020 7:54 AM, deloptes wrote:
> Stefan Monnier wrote:
>
>> What makes you think so?  Does it overheat?  Is the "load average" too
>> high? Where is the CPU time spent (i.e. e.g. what does `top` say, is it
>> mostly in "wait"?  "kernel"?  "user"?)?
>>

Load avrage as shown by 'uptime' and my CPU was at full speed when using
the usbkey ('lscpu').

>> Have you tried to install a similar (tho probably smaller) VM on your
>> internal storage just in order to see if the problem comes from the
>> external storage or from the VM itself?
>

I can confirm that the issue is only when using the usbkey (external
storage).

> IMO, allthose question as irrelevant - the problem is the IO. You can try it
> easily. get USB 2.0 and put a VM on it - try to start it - you will know
> what OP means. Obviously also USB3 are not always USB3 :D

Thanks to DSR, I know that I have a slow device (lacking 'UASP' support)
and this is most likely the issue as everything works fine when using
internal storage.
As uptime takes IO into consideration, what's the best way to
demonstrate that fact?


In other words, what command can I use to ensure that it is a IO issue.


To Deloptes,

I have a single board with only USB 3.0 but thanks for your thoughts.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#225733

[Multipart message — attachments visible in raw view] — view raw

On Vi, 31 iul 20, 10:32:21, john doe wrote:
> 
> In other words, what command can I use to ensure that it is a IO issue.

Try 'iotop', in the package of the same name.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#225615

> IMO, allthose question as irrelevant - the problem is the IO. You can try it
> easily. get USB 2.0 and put a VM on it -

I have and have had no such problems.
I strongly suspect that there's something else at play.
E.g. its VM is performing a lot of disk IO.


        Stefan

[toc] | [prev] | [next] | [standalone]

#225668

Stefan Monnier wrote:

> I have and have had no such problems.
> I strongly suspect that there's something else at play.
> E.g. its VM is performing a lot of disk IO.

You mean you can run a VM from USB2.0 smoothly?

[toc] | [prev] | [next] | [standalone]

#225765

>> I have and have had no such problems.
>> I strongly suspect that there's something else at play.
>> E.g. its VM is performing a lot of disk IO.
> You mean you can run a VM from USB2.0 smoothly?

Yes, it works just like from the internal drive (just a bit less
efficient, just like all other non-VM disk accesses a less efficient
when going over USB).


        Stefan

[toc] | [prev] | [next] | [standalone]

#225679

On 7/31/2020 2:52 PM, Stefan Monnier wrote:
>> IMO, allthose question as irrelevant - the problem is the IO. You can try it
>> easily. get USB 2.0 and put a VM on it -
>
> I have and have had no such problems.
> I strongly suspect that there's something else at play.
> E.g. its VM is performing a lot of disk IO.
>

Can you expand on your thoughts?

The issue only arises when the usbkey is used.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#225683

[Multipart message — attachments visible in raw view] — view raw

That's the problem: An internal SSD is connected relatively directly to the
CPU. Almost all USB keys aren't optimised for fast data transfer other
than, perhaps, reading/buffering large  single files. Running a VM means
constant read/write, constant update - it would be all the same as pushing
a CPU to constant use of a swap file/swap partition. It's massively
stressful on I/O - if you've a USB 2.x device, it transfers (very) slowly
compared to USB 3.x.

[For an illustration: even identical USB sticks will also vary: writing
2.7GB DVD images to a set of USB 2.0 sticks a couple of weeks ago: most of
them took six minutes, one took 35 - poor i/o and constantly swapping to
check (I used dd with the oflag=sync option - essential for data
integrity.) ]

The nearest comparison would be using a nice new NVME device. Inside a
laptop, in an M.2 slot - as fast as you like. Put it into an external USB
connected caddy - via USB 3.1 - for data transfer to another NVME inside
the laptop and it will be slightly to significantly slower. Same chip, same
laptop, moderated through a slower connection. That's the difference
between your VM on internal disk and your VM on USB stick

On Sat, Aug 1, 2020 at 8:44 AM john doe <johndoe65534@mail.com> wrote:

> On 7/31/2020 2:52 PM, Stefan Monnier wrote:
> >> IMO, allthose question as irrelevant - the problem is the IO. You can
> try it
> >> easily. get USB 2.0 and put a VM on it -
> >
> > I have and have had no such problems.
> > I strongly suspect that there's something else at play.
> > E.g. its VM is performing a lot of disk IO.
> >
>
> Can you expand on your thoughts?
>
> The issue only arises when the usbkey is used.
>
> --
> John Doe
>
>

[toc] | [prev] | [next] | [standalone]

#225685

On 8/1/2020 11:44 AM, Andrew Cater wrote:
> That's the problem: An internal SSD is connected relatively directly to the
> CPU. Almost all USB keys aren't optimised for fast data transfer other
> than, perhaps, reading/buffering large  single files. Running a VM means
> constant read/write, constant update - it would be all the same as pushing
> a CPU to constant use of a swap file/swap partition. It's massively
> stressful on I/O - if you've a USB 2.x device, it transfers (very) slowly
> compared to USB 3.x.

I have an USB 3.0 (USb port and USB key).

>
> [For an illustration: even identical USB sticks will also vary: writing
> 2.7GB DVD images to a set of USB 2.0 sticks a couple of weeks ago: most of
> them took six minutes, one took 35 - poor i/o and constantly swapping to
> check (I used dd with the oflag=sync option - essential for data
> integrity.) ]
>
> The nearest comparison would be using a nice new NVME device. Inside a
> laptop, in an M.2 slot - as fast as you like. Put it into an external USB
> connected caddy - via USB 3.1 - for data transfer to another NVME inside
> the laptop and it will be slightly to significantly slower. Same chip, same
> laptop, moderated through a slower connection. That's the difference
> between your VM on internal disk and your VM on USB stick
>

Okay, given that I can't increase the internal sise, my only way out is
an USB 3.0 that supports UASP or mounting a remote share by using sshfs...

I had understood Stefan's comment that it could be something else as well.

--
John Doe

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web