Re: Speed ups for a disk IO bound machine

From	Peter Chant <pete@petezilla.co.uk>
Newsgroups	comp.os.linux.hardware
Subject	Re: Speed ups for a disk IO bound machine
Date	2016-12-15 23:18 +0000
Message-ID	<ebgmmmFbr72U1@mid.individual.net> (permalink)
References	(4 earlier) <bcrvhdxjip.ln2@phoenix.vfire> <6rvvhd-1d5.ln1@lazy.lzy> <o2oe99$23j$1@dont-email.me> <ebdmriFk19eU1@mid.individual.net> <o2u094$s1f$1@dont-email.me>

Show all headers | View raw

On 12/15/2016 11:55 AM, David Brown wrote:
> On 14/12/16 21:03, Peter Chant wrote:
>> On 12/13/2016 09:17 AM, David Brown wrote:
>>
>>>>> OK, so having done some reading up but not carried out any testing I
>>>>> think the following is a possible setup.  Note I am using LUKS so I will
>>>>> add this extra layer to the mix.  I have not noticed any significant
>>>>> performance difference with and without it.
>>>
>>> This makes me even more suspicious that you (the OP) really have an I/O
>>> problem, or have identified where it is.
>>>
>>
>>
>> I've had periods where the disks have been solidly at 80-90% utalisation
>> for many seconds yet the CPU has been lightly loaded.
> 
> What is the computer doing at the time?
> 

Can't remember specifically I'm afraid.


> Is this actually slowing down something that you are waiting for?
> 

It has annoyed me as I was doing other stuff.


> It is /normal/ for a computer to run at maximum in some aspects, for
> some time.  If you are transferring a large file, you /want/ the disks
> to be as close to 100% as possible.  If you are doing a raid1 scrub, you
> /want/ the disks to be close to 100%, perhaps for hours at a time if the
> disks are big (but at low I/O priority so that other tasks can also run).
> 
> So far, you have just told me that your system is working.
> 

Well, yes.  But when I've not been deduping etc so it is not expected it
can get annoying.



>> Well.  If raiding SSD's there is some logic to this horrible looking
>> asymetric setup.  I'm using an SSD for the OS and that has plenty of
>> free space.  I've also the older smaller SSD it replaced.  Although it
>> looks messy I could free up some space on the current system SSD to use
>> a partition for RAID and use that in combination with the old, currently
>> unused SSD.  That saves shelling out for another SSD if I want to RAID a
>> pair.  Anaesthetically it is horrid and obviously would impact the speed
>> of OS access.  But is it hardware I have so the monetary cost is no
>> issue.  The time and hair loss cost might not be so trivial.
> 
> OK - my first thought was that it was a mistake (possibly just a missing
> character from a cut-and-paste).  Now I see it is intentional.
> 

It was a crude example to see if I had understood the basic concept, so
I was not certain I was 100%, just mainly right.

> I am not convinced this will be a good thing - in fact, I am confident
> that it is a /bad/ thing.  An old small SSD can easily be a /lot/ slower
> than a new one - typically, old and small SSDs have poor garbage
> collection and little over-provisioning.  This means they get very slow
> at writes when they are full - even slower, sometimes, than hard disks.
>  When you create an md raid1 array like this, the first thing md will do
> is copy block-for-block from /dev/sda4 to /dev/sdb.  It will write to
> the entire disk - the old SSD will think that /all/ its normal blocks
> are full of important data.  As soon as you try to write something else
> to it, it must now try to do garbage collection in "panic" mode with
> minimal free space - you might find your write latencies measured in
> /seconds/.  And if the SSD is old enough, then it won't be able to
> handle reads during the erases and garbage collection.  If you are
> lucky, reads will come from the other disk.  If you are unlucky, reads
> will be stalled too.
> 
> So I would expect your system to be a good deal slower by doing this,
> compared to simply using the new SSD on its own.

Well, the old SSD is not that old its a Samsung 840 Pro. Anyway.  I have
it and am not otherwise using it and the same goes for a 2TB HDD. So I
can play with caching (but not necessarily RAID).  So I can add them to
the machine and have a go.  However, this is much larger than the
mariadb files and some others so I can try using it as a dedicated drive
for some things and see if shifting some storage to it makes a
difference.  With more effort I can probably play with RAID and caching.

I've also looked up mysql optimisation and xfs is suggested.  So I can
make a dedicated xfs partition and see what happens.  With or without LVM...

There seem to be two schools of thought on RAID1 here.  Anyway, I can
play as far as I like provided I don't waste time or money.

> 
> /If/ your old SSD is reasonably fast, you can first run a secure erase
> to tell it to drop all data.  Then partition it, leaving a little extra
> space unused - this overprovisioning can help a lot.  And use it with
> btrfs raid1, not md raid1, so that only the actual useful data is
> replicated.
> 

Think I'll boot on a USB stick and unplug all other drives when trying that.


>>
>> Would image the SSD in case of mess ups before resizing partitions.
>>
>> As for the whole of /dev/sdb - I've been using btrfs for a while and it
>> is normal to give it whole disks, a simple slip of the finger.
> 
> I always use partitions, but I usually want a couple of partitions for
> other things (like swap).
> 
>>
>>> I don't think anything has been said about the sizes and partitioning of
>>> the SSD.  For smaller or cheaper SSDs, it is worth leaving a small
>>> amount of unpartitioned space at the end of the disk to give it more
>>> flexibility in garbage collection.  (Do a secure erase before
>>> partitioning if it is not a new clean SSD.)
>>>
>>
>> Have done that already.
> 
> Including the unused space at the end?

Yes. (but not on the old drive so back to the secure erase step).

> 
>>
>>>>
>>>> Metadata 1.0, 1.1 and 1.2 are the new ones, use these,
>>>> not the 0.90, which have less features.
>>>>
>>>> Maybe, not really sure, but partitioning /dev/sdb might
>>>> be better.
>>>>
>>>>> Use crypt setup to create an encrypted block device on top of this raid
>>>>> 10 far 2 device.
>>>>>
>>>>> Use LVM to create a volume group on top of the encrypted raid 10 far 2
>>>>> device.
>>>>
>>>> Or the other way around.
>>>> I'm not sure which is better, maybe your proposal.
>>>
>>> Neither is better, IMHO, unless you have some reason to be seriously
>>> paranoid.  It is understandable why one would want to encrypt a portable
>>> machine that you use a lot while travelling, but a home desktop?  Think
>>> about whether encryption here is really a useful thing - adding layers
>>> of complexity does not make anything faster, and it makes it a whole lot
>>> more difficult if something goes wrong or if you want to recover your
>>> files from a different system.
>>>
>>
>> Well, in this day and age it seemed like a reasonable idea.
> 
> It can be fun to play around with this sort of thing, but that doesn't
> mean it is a good idea if you are aiming for a useful system.
> 
> One particular thing that can be a serious pain with such complex setups
> is if something goes wrong.  If something breaks badly, you might have
> to put the disks in a different machine to recover the data, or boot the
> same machine from a live USB.  If you have a lvm cache or bcache in
> writeback mode and you have both the SSD and the HD online, and you may
> need a system with the right kernel and utilities in order to get the
> disks working.  Even then, it's easy to accidentally corrupt things
> along the way such as by writing to the HD while there are uncommitted
> changes on the SSD part.  I have enough experience with complicated
> recoveries to know that when something goes wrong, you'll be glad you
> kept things simple - and that you documented everything :-)
> 

The system is on a simple partition with btrfs in single.  But boot is
ext3 or 4.  And there is a partition I consider the 'maintenance'
partition with a full slackware install on ext3 or 4 that I hardly touch
but is handy in case I break something.  Also you can't build a kernel
from a rescue disk but you can with this which is nice.

You can reasonably argue that it is a waste of space especially on an
SSD which is expensive compared to a HDD but it is a nice comfort
blanket and doubly so on the rare occasions it is needed.



>>
>>>>
>>>>> Create a logical volume on top of the above to use as the cache device.
>>>>
>>>> Yep, if you use lvmcache, maybe bcache can do as well.
>>>
>>> Again, that's unnecessary complexity for a system like this.  The
>>> benefits of lvmcache and bcache are debatable even for loads that match
>>> them.
>>>
>>
>> So it is about as fast as it will get now?
>>
>> I've a spare hdd and ssd so I'll have a play when I get time.  Need to
>> think about a useful benchmark.
> 
> It's all about the type of load you are using.  lvmcache and/or bcache
> can help a lot in some cases - but be no faster than a single HD in
> other cases.  And for many of the things that /do/ run faster with the
> caches, you could just as easily run them entirely from the SSD
> significantly faster - or get even better results by adding more RAM.
> 

Well.  I have the options.  What I'm not going to do is by a blister
pack of expensive terrabyte range SSD for everything.

> The key to getting /really/ optimal systems is, as you say, useful
> benchmarks.  There is no benefit in looking at the timings of a test
> that writes lots of small blocks from lots of threads unless that really
> is what you are doing in practice.  There is no benefit in running a
> benchmark after clearing your caches, because you don't clear your
> caches in practice - but if you run without clearing the caches, you are
> not testing your disk performance.  The only "true" benchmark is to run
> your system with your typical real tasks and see how it performs.
> 
>>
>>
>>

>>> You only want the raid1 at one level.  Your choice is raid1 on btrfs for
>>> best performance and efficiency (since only the actual useful data is
>>> mirrored, rather than the entire raw disk), or raid10,far on the md
>>> layer (for greater large file streaming read speed).  This can be a big
>>> issue with SSDs - with btrfs raid1 you avoid initially copying over an
>>> entire diskful of data from one device to the other.
>>
>> replacing a disk and the associated balance took a week.
> 
> If you have a lot of data, it takes a while to copy it all over.  And a
> re-balance actually does a good deal more work than just a plain copy.
> But at least it doesn't copy the unused space too.
> 

Ues, there is the checksumming for a start.  But that is a good thing.




> 
> Loop devices work well in testing, especially for seeing how to add
> disks, replace disks, resize things, etc.  Of course they are of little
> help in speed testing.
> 
> I usually testing using loop devices made in a tmpfs mount - but then,
> my machines normally have lots of ram.
> 

I'd never have guessed.  :-)


Anyway food for thought here from Piergorgio and yourself.

Thread

Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-07 22:41 +0000
  Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-08 16:44 +0000
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 11:24 +0000
      Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-11 01:34 +0000
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:07 +0000
  Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 12:43 +0100
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 20:44 +0000
      Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 23:05 +0100
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 11:54 +0000
          Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-11 14:10 +0100
            Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:46 +0000
              Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-12 19:22 +0100
            Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 10:17 +0100
              Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 20:03 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 12:55 +0100
                Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 23:18 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:34 +0100
        Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:57 +0100
  Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-12 09:11 +0100
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-12 23:08 +0000
      Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:29 +0100
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 23:16 +0000
          Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 13:15 +0100
            Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 22:50 +0000
              Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:39 +0100

csiph-web