Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.hardware > #3261

Re: Speed ups for a disk IO bound machine

Show all headers | View raw

On 14/12/16 21:03, Peter Chant wrote:
> On 12/13/2016 09:17 AM, David Brown wrote:
> 
>>>> OK, so having done some reading up but not carried out any testing I
>>>> think the following is a possible setup.  Note I am using LUKS so I will
>>>> add this extra layer to the mix.  I have not noticed any significant
>>>> performance difference with and without it.
>>
>> This makes me even more suspicious that you (the OP) really have an I/O
>> problem, or have identified where it is.
>>
> 
> 
> I've had periods where the disks have been solidly at 80-90% utalisation
> for many seconds yet the CPU has been lightly loaded.

What is the computer doing at the time?

Is this actually slowing down something that you are waiting for?

It is /normal/ for a computer to run at maximum in some aspects, for
some time.  If you are transferring a large file, you /want/ the disks
to be as close to 100% as possible.  If you are doing a raid1 scrub, you
/want/ the disks to be close to 100%, perhaps for hours at a time if the
disks are big (but at low I/O priority so that other tasks can also run).

So far, you have just told me that your system is working.

> 
> 
>>>> Create raid 10 far 2 raid array with mdadm for example (need to check
>>>> what metadata means!):
>>>> mdadm --create /dev/md-ssd --level=10 --metadata=0.90 --raid-devices=2
>>>> --layout=f2 /dev/sda4 /dev/sdb
>>
>> That looks like you are raiding a partition on one device with the
>> entire second device.  Are you sure that is what you want?
>>
> 
> Well.  If raiding SSD's there is some logic to this horrible looking
> asymetric setup.  I'm using an SSD for the OS and that has plenty of
> free space.  I've also the older smaller SSD it replaced.  Although it
> looks messy I could free up some space on the current system SSD to use
> a partition for RAID and use that in combination with the old, currently
> unused SSD.  That saves shelling out for another SSD if I want to RAID a
> pair.  Anaesthetically it is horrid and obviously would impact the speed
> of OS access.  But is it hardware I have so the monetary cost is no
> issue.  The time and hair loss cost might not be so trivial.

OK - my first thought was that it was a mistake (possibly just a missing
character from a cut-and-paste).  Now I see it is intentional.

I am not convinced this will be a good thing - in fact, I am confident
that it is a /bad/ thing.  An old small SSD can easily be a /lot/ slower
than a new one - typically, old and small SSDs have poor garbage
collection and little over-provisioning.  This means they get very slow
at writes when they are full - even slower, sometimes, than hard disks.
 When you create an md raid1 array like this, the first thing md will do
is copy block-for-block from /dev/sda4 to /dev/sdb.  It will write to
the entire disk - the old SSD will think that /all/ its normal blocks
are full of important data.  As soon as you try to write something else
to it, it must now try to do garbage collection in "panic" mode with
minimal free space - you might find your write latencies measured in
/seconds/.  And if the SSD is old enough, then it won't be able to
handle reads during the erases and garbage collection.  If you are
lucky, reads will come from the other disk.  If you are unlucky, reads
will be stalled too.

So I would expect your system to be a good deal slower by doing this,
compared to simply using the new SSD on its own.

/If/ your old SSD is reasonably fast, you can first run a secure erase
to tell it to drop all data.  Then partition it, leaving a little extra
space unused - this overprovisioning can help a lot.  And use it with
btrfs raid1, not md raid1, so that only the actual useful data is
replicated.

> 
> Would image the SSD in case of mess ups before resizing partitions.
> 
> As for the whole of /dev/sdb - I've been using btrfs for a while and it
> is normal to give it whole disks, a simple slip of the finger.

I always use partitions, but I usually want a couple of partitions for
other things (like swap).

> 
>> I don't think anything has been said about the sizes and partitioning of
>> the SSD.  For smaller or cheaper SSDs, it is worth leaving a small
>> amount of unpartitioned space at the end of the disk to give it more
>> flexibility in garbage collection.  (Do a secure erase before
>> partitioning if it is not a new clean SSD.)
>>
> 
> Have done that already.

Including the unused space at the end?

> 
>>>
>>> Metadata 1.0, 1.1 and 1.2 are the new ones, use these,
>>> not the 0.90, which have less features.
>>>
>>> Maybe, not really sure, but partitioning /dev/sdb might
>>> be better.
>>>
>>>> Use crypt setup to create an encrypted block device on top of this raid
>>>> 10 far 2 device.
>>>>
>>>> Use LVM to create a volume group on top of the encrypted raid 10 far 2
>>>> device.
>>>
>>> Or the other way around.
>>> I'm not sure which is better, maybe your proposal.
>>
>> Neither is better, IMHO, unless you have some reason to be seriously
>> paranoid.  It is understandable why one would want to encrypt a portable
>> machine that you use a lot while travelling, but a home desktop?  Think
>> about whether encryption here is really a useful thing - adding layers
>> of complexity does not make anything faster, and it makes it a whole lot
>> more difficult if something goes wrong or if you want to recover your
>> files from a different system.
>>
> 
> Well, in this day and age it seemed like a reasonable idea.

It can be fun to play around with this sort of thing, but that doesn't
mean it is a good idea if you are aiming for a useful system.

One particular thing that can be a serious pain with such complex setups
is if something goes wrong.  If something breaks badly, you might have
to put the disks in a different machine to recover the data, or boot the
same machine from a live USB.  If you have a lvm cache or bcache in
writeback mode and you have both the SSD and the HD online, and you may
need a system with the right kernel and utilities in order to get the
disks working.  Even then, it's easy to accidentally corrupt things
along the way such as by writing to the HD while there are uncommitted
changes on the SSD part.  I have enough experience with complicated
recoveries to know that when something goes wrong, you'll be glad you
kept things simple - and that you documented everything :-)

> 
>>>
>>>> Create a logical volume on top of the above to use as the cache device.
>>>
>>> Yep, if you use lvmcache, maybe bcache can do as well.
>>
>> Again, that's unnecessary complexity for a system like this.  The
>> benefits of lvmcache and bcache are debatable even for loads that match
>> them.
>>
> 
> So it is about as fast as it will get now?
> 
> I've a spare hdd and ssd so I'll have a play when I get time.  Need to
> think about a useful benchmark.

It's all about the type of load you are using.  lvmcache and/or bcache
can help a lot in some cases - but be no faster than a single HD in
other cases.  And for many of the things that /do/ run faster with the
caches, you could just as easily run them entirely from the SSD
significantly faster - or get even better results by adding more RAM.

The key to getting /really/ optimal systems is, as you say, useful
benchmarks.  There is no benefit in looking at the timings of a test
that writes lots of small blocks from lots of threads unless that really
is what you are doing in practice.  There is no benefit in running a
benchmark after clearing your caches, because you don't clear your
caches in practice - but if you run without clearing the caches, you are
not testing your disk performance.  The only "true" benchmark is to run
your system with your typical real tasks and see how it performs.

> 
> 
> 
>>
>> btrfs does not have raid10, and it does not share significant raid1 or
>> raid0 code with md.  (It /does/ share code for calculating raid5 and
>> raid6 parities, but that's another issue - and don't use btrfs raid5/6
>> until the bugs are sorted out!).
> 
> Oh yes it does. :-).  But I don't see anything about far 2.

Btrfs "raid10" is (roughly) traditional raid1+0, where you have two
mirrors which are striped.  (I say roughly, because btrfs handles this
at the level of extents, rather than raw disk blocks.)  Linux md raid10
is quite a bit more sophisticated than just layered raid1 and raid0 -
which is why it works on any number of disks, not just multiples of 4.

> 
>>
>> You only want the raid1 at one level.  Your choice is raid1 on btrfs for
>> best performance and efficiency (since only the actual useful data is
>> mirrored, rather than the entire raw disk), or raid10,far on the md
>> layer (for greater large file streaming read speed).  This can be a big
>> issue with SSDs - with btrfs raid1 you avoid initially copying over an
>> entire diskful of data from one device to the other.
> 
> replacing a disk and the associated balance took a week.

If you have a lot of data, it takes a while to copy it all over.  And a
re-balance actually does a good deal more work than just a plain copy.
But at least it doesn't copy the unused space too.

> 
> 
> I'm at RAID1 with btrfs now.  Yes, not RAIDing btrfs over another RAID
> as that make little sense here.
> 
>>>
>>>> Of course, I could have got completely the wrong idea above and invented
>>>> some horrid monster!
>>>
>>> If I understood it right, it sounds OK to me.
>>>
>>> I would, in any case, strongly suggest to experiment,
>>> maybe, as mentioned above, with loop devices.
>>
>> Agreed.
> 
> Spare hdd and ssd. Could use loop devices but perhaps real hw is useful,
> though not enough to RAID.

Loop devices work well in testing, especially for seeing how to add
disks, replace disks, resize things, etc.  Of course they are of little
help in speed testing.

I usually testing using loop devices made in a tmpfs mount - but then,
my machines normally have lots of ram.

> 
>>
>>> Not for performances, but for practising possible
> 

Back to comp.os.linux.hardware | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-07 22:41 +0000
  Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-08 16:44 +0000
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 11:24 +0000
      Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-11 01:34 +0000
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:07 +0000
  Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 12:43 +0100
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 20:44 +0000
      Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 23:05 +0100
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 11:54 +0000
          Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-11 14:10 +0100
            Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:46 +0000
              Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-12 19:22 +0100
            Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 10:17 +0100
              Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 20:03 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 12:55 +0100
                Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 23:18 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:34 +0100
        Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:57 +0100
  Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-12 09:11 +0100
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-12 23:08 +0000
      Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:29 +0100
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 23:16 +0000
          Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 13:15 +0100
            Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 22:50 +0000
              Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:39 +0100

csiph-web