Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.hardware > #3258

Re: Speed ups for a disk IO bound machine

Show all headers | View raw

On 11/12/16 14:10, Piergiorgio Sartor wrote:
> On 2016-12-11 12:54, Peter Chant wrote:
> [...]
>> Thanks.  I've done some reading and there is more to do plus some
>> experimentation.
> 
> Experimentation is good.
> It is the only way to have so idea of the
> different peculiarities.
> 
>> I understand that mdadm is used to create the raid arrays, it is not
>> part of lvm itself?
> 
> LVM (dmraid) and md share a lot of code, but I'm
> not sure about this RAID-10. Maybe it is only md.
> 

lvm can do raid1 and raid0, and can be layered to give a traditional
4-drive raid10.  But it cannot do md-style raid10, nor the other more
advanced raid setups.  It is also less efficient than md raid - but
sometimes more convenient because it is much easier to have the raid
settings change on the fly (such as if you try to make a raid0 stripe
from physical volumes of different sizes - lvm will stripe as much as
possible, and give you unstriped use of the rest of the space).

> [...]
>> OK.  Simple to set up with the SSDs as one is blank and the other has
>> free space.
> 
> Be careful, it is easy to destroy data.
> Maybe you can practice with loop devices.
> There are some howtos around.
> 
>> With RAID-10 far 2 am I correct in assuming that the available capacity
>> for a two device array of 3TB disks would be 3TB (two copies of data)?
> 
> Yep.
> 
>>> Part of the RAID-10 with SSD can be used as cache.
>>>
>>
>> OK, so having done some reading up but not carried out any testing I
>> think the following is a possible setup.  Note I am using LUKS so I will
>> add this extra layer to the mix.  I have not noticed any significant
>> performance difference with and without it.

This makes me even more suspicious that you (the OP) really have an I/O
problem, or have identified where it is.

> 
> OK, you'll have to decide at which layer LUKS fits.
> There are pro and cons for each case.
> 
>> SSDs:
>> Create empty partition on larger system SSD.
>> Add smaller empty SSD.
>>
>> Create raid 10 far 2 raid array with mdadm for example (need to check
>> what metadata means!):
>> mdadm --create /dev/md-ssd --level=10 --metadata=0.90 --raid-devices=2
>> --layout=f2 /dev/sda4 /dev/sdb

That looks like you are raiding a partition on one device with the
entire second device.  Are you sure that is what you want?

I don't think anything has been said about the sizes and partitioning of
the SSD.  For smaller or cheaper SSDs, it is worth leaving a small
amount of unpartitioned space at the end of the disk to give it more
flexibility in garbage collection.  (Do a secure erase before
partitioning if it is not a new clean SSD.)

> 
> Metadata 1.0, 1.1 and 1.2 are the new ones, use these,
> not the 0.90, which have less features.
> 
> Maybe, not really sure, but partitioning /dev/sdb might
> be better.
> 
>> Use crypt setup to create an encrypted block device on top of this raid
>> 10 far 2 device.
>>
>> Use LVM to create a volume group on top of the encrypted raid 10 far 2
>> device.
> 
> Or the other way around.
> I'm not sure which is better, maybe your proposal.

Neither is better, IMHO, unless you have some reason to be seriously
paranoid.  It is understandable why one would want to encrypt a portable
machine that you use a lot while travelling, but a home desktop?  Think
about whether encryption here is really a useful thing - adding layers
of complexity does not make anything faster, and it makes it a whole lot
more difficult if something goes wrong or if you want to recover your
files from a different system.

> 
>> Create a logical volume on top of the above to use as the cache device.
> 
> Yep, if you use lvmcache, maybe bcache can do as well.

Again, that's unnecessary complexity for a system like this.  The
benefits of lvmcache and bcache are debatable even for loads that match
them.

> 
>> HDD:
>> Similar to the SSD case above except that the logical volume will be for
>> the slow disks doing the bulk of the work.
> 
> OK, I think.
> 
>> Create a cached device:
>> Use lvmcache to create a device using the the two logical volumes
>> created above (bcache would also work).
> 
> Seems good to me.
> 
>> Create a file system on top of the cached device.  If using btrfs (what
>> I do now) use single for data as the raid is occuring a couple of levels
>> down.
> 
> Well, it might be btrfs has already RAID-10.
> Again, code is shared between this and md too.

btrfs does not have raid10, and it does not share significant raid1 or
raid0 code with md.  (It /does/ share code for calculating raid5 and
raid6 parities, but that's another issue - and don't use btrfs raid5/6
until the bugs are sorted out!).

You only want the raid1 at one level.  Your choice is raid1 on btrfs for
best performance and efficiency (since only the actual useful data is
mirrored, rather than the entire raw disk), or raid10,far on the md
layer (for greater large file streaming read speed).  This can be a big
issue with SSDs - with btrfs raid1 you avoid initially copying over an
entire diskful of data from one device to the other.

> 
>> Of course, I could have got completely the wrong idea above and invented
>> some horrid monster!
> 
> If I understood it right, it sounds OK to me.
> 
> I would, in any case, strongly suggest to experiment,
> maybe, as mentioned above, with loop devices.

Agreed.

> Not for performances, but for practising possible
> combinations and layouts.

Agreed.

> 
> Then there is the story of the caching, which has
> different scope and performances between bcache
> and lvmcache.
> In your specific case, I cannot judge which is better,
> but lvmcache seems to me easier.
> 
>> It looks like I could do this without the LVM layer between the LUKs and
>> cache parts.  However, this does give me the flexability to create
>> logical volumes that are on HDD RAID10, SDD RAID10 soley as well as
>> cached.  Or different cache options.  I think I could have put another
>> LVM on top of the cached item I was creating but I think that was over
>> doing it.
> 
> Probably it is.
> I would try to use not more than one component at time.
> So, 1 md, 1 LUKS, 1 LVM, at maximum.
> If possible less.
> 
> It would be also possible to create two PVs out of the two
> RAID-10 and a single VG with both.
> Then the LV can be fitted in one or the other PV.
> LUKS can be at RAID level or on top of LV.
> 
> This type of "generic" setup has some advantages in case
> of hardware updates (easy to add / remove storage devices,
> by using pvmove).
> 
> bye,
> 

Back to comp.os.linux.hardware | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-07 22:41 +0000
  Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-08 16:44 +0000
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 11:24 +0000
      Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-11 01:34 +0000
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:07 +0000
  Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 12:43 +0100
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 20:44 +0000
      Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 23:05 +0100
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 11:54 +0000
          Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-11 14:10 +0100
            Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:46 +0000
              Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-12 19:22 +0100
            Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 10:17 +0100
              Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 20:03 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 12:55 +0100
                Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 23:18 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:34 +0100
        Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:57 +0100
  Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-12 09:11 +0100
    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-12 23:08 +0000
      Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:29 +0100
        Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 23:16 +0000
          Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 13:15 +0100
            Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 22:50 +0000
              Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:39 +0100

csiph-web