Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #30809

Re: Multithreaded disk access

Show all headers | View raw

On 10/15/2021 9:48 AM, Richard Damon wrote:
> On 10/15/21 12:00 PM, Don Y wrote:
>> On 10/15/2021 8:38 AM, Richard Damon wrote:
>>> You know that you can't actually get data off the media faster than the 
>>> fundamental data rate of the media.
>>
>> Yes, but you don't know that rate *and* that rate varies based on
>> "where" you're accesses land on the physical medium (e.g., ZDR,
>> shingled drives, etc.)
> 
> But all of these still have a 'maximum' rate, so you can still define a 
> maximum. It does say that the 'expected' rate you can get gets more variable.

But only if you have control over the hardware.

How long will a "backup" take on your PC.  Today?
Tomorrow?  Last week?

If you removed the disk and put it in another PC,
how would those figures change?

If you restore (using file access and not sector access),
and then backup again, how will the numbers change?

>>> As you mention, cache can give an apparent rate faster than the media, but 
>>> you seem to be willing to assume that caching doesn't affect your rate, and 
>>> each chunk will only be returned once.
>>
>> Cache in the filesystem code will be counterproductive.  Cache in
>> the drive may be a win for some accesses and a loss for others
>> (e.g., if the drive read ahead thinking the next read was going to
>> be sequential with the last -- and that proves to be wrong -- the
>> drive may have missed an opportunity to respond more quickly to the
>> ACTUAL access that follows).
>>
>> [I'm avoiding talking about reads AND writes just to keep the
>> discussion complexity manageable -- to avoid having to introduce
>> caveats with every statement]
> 
> Yes, the drive might try to read ahead and hurt itself, or it might not. That 
> is mostly out of your control.

Exactly.  So, I can't do anything other than OBSERVE the performance
I am getting.

>>> Non-Pathological access can often result in near 100% of the access rate.
>>>
>>> The best test of if you are I/O bound is if the I/O system is constantly in 
>>> use, and every I/O request has another pending when it finishes, then you 
>>> are totally I/O bound.
>>
>> But, if you make that assessment when the access pattern is "unfortunate",
>> you erroneously conclude the disk is at its capacity.  And, vice versa.
>>
>> Without control over the access patterns, it seems like there is no
>> reliable strategy for determining when another thread can be
>> advantageous (?)
> 
> Yes, adding more threads might change the access pattern. it will TEND to make 
> the pattern less sequential, and thus more towards that pathological case (and 
> thus more threads actually decrease the rate you can do I/O and thus slow down 
> your I//O bound rate). It is possible that it just happens to be fortunate to 
> make things more sequential, if the system can see that one thread wants sector 
> N and another wants sector N+1, something can schedule the reads together and 
> drop a seek.

The point of additional threads is that another thread can schedule
the next access while the processor is busy processing the previous
one.  So, the I/O is always kept busy instead of letting it idle
between accesses.

> Predicting that sort of behavior can't be done 'in the abstract'. You need to 
> think about the details of the system.
> 
> As a general principle, if the I/O system is saturated, the job is I/O bound. 

The goal is to *ensure* the I/O system is completely saturated.

> Adding more threads will only help if you have the resources to queue up more 
> requests and can optimize the order of servicing them to be more efficient with 

Ordering them is an optimization that requires knowledge of how they
will interact *in* the drive.  However, simply having ANOTHER request
ready as soon as the previous one is completed (neglecting the
potential for the drive to queue requests internally) is an
enhancement to throughput.

> I/O. Predicting that means you need to know and have some control over the 
> access pattern.
> 
> Note, part of this is being able to trade memory to improve I/O speed. If you 
> know that EVENTUALLY you will want the next sector after the one you are 
> reading, reading that now and caching it will be a win, but only if you will be 
> able to use that data before you need to claim that memory for other uses. This 
> sort of improvement really does require knowing details you want to try to 
> assume you don't want to know, so you are limiting your ability to make 
> accurate decisions.

Moving the code to another platform (something that the user can do
in a heartbeat) will invalidate any assumptions I have made about the
performance on my original platform.  Hence the desire to have the
*code* sort out what it *can* do to increase performance by observing
its actual performance on TODAY'S actual hardware.

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 07:08 -0700
  Re: Multithreaded disk access Richard Damon <Richard@Damon-Family.org> - 2021-10-15 11:38 -0400
    Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 09:00 -0700
      Re: Multithreaded disk access Richard Damon <Richard@Damon-Family.org> - 2021-10-15 12:48 -0400
        Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 11:40 -0700
  Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-15 18:46 +0300
    Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 09:19 -0700
      Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-15 19:28 +0300
  Re: Multithreaded disk access Brett <ggtgp@yahoo.com> - 2021-10-17 20:27 +0000
    Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-17 14:49 -0700
      Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-17 15:01 -0700
      Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-18 01:09 +0300
        Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-18 08:05 -0700
          Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-18 19:25 +0300
            Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-18 13:46 -0700
              Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-19 00:46 +0300
                Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-18 16:17 -0700
                Re: Multithreaded disk access antispam@math.uni.wroc.pl - 2021-10-31 19:21 +0000
                Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-31 21:55 +0200
              Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-25 04:19 -0700
                Re: Multithreaded disk access Richard Damon <Richard@Damon-Family.org> - 2021-10-25 20:23 -0400
                Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-25 22:29 -0700
      Re: Multithreaded disk access Clifford Heath <no.spam@please.net> - 2021-10-18 11:58 +1100

csiph-web