Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #30804

Re: Multithreaded disk access

Show all headers | View raw

On 10/15/2021 17:08, Don Y wrote:
> As a *rough* figure, what would you expect the bandwidth of
> a disk drive (spinning rust) to do as a function of number of
> discrete files being accessed, concurrently?
> 
> E.g., if you can monitor the rough throughput of each
> stream and sum them, will they sum to 100% of the drive's
> bandwidth?  90%?  110?  etc.
> 
> [Note that drives have read-ahead and write caches so
> the speed of the media might not bleed through to the
> application layer.  And, filesystem code also throws
> a wrench in the works.  Assume caching in the system
> is disabled/ineffective.]

If caching is disabled things can get really bad quite quickly,
think on updating directory entries to reflect modification/access
dates, file sizes, scattering etc., think also allocation
table accesses etc. E.g. in dps on a larger disk partition
(say >100 gigabytes) the first CAT (cluster allocation table)
access after boot takes some noticeable time, a second maybe;
then it stops being noticeable at all as the CAT is updated
rarely and on a modified area basis only (this on a a processor
capable of 20 Mbytes/second) (dps needs the entire CAT to allocate
new space in order to do its (enhanced) worst fit scheme).
IOW if you torture the disk with constant seeks and scattered
accesses you can slow it down from somewhat to a lot, depends
on way too many factors to be worth wondering about.

> 
> Said another way, what's a reasonably reliable way of
> determining when you are I/O bound by the hardware
> and when more threads won't result in more performance?

Just try it out for some time and make your pick. Recently
I did that dfs (distributed file system, over tcp) for dps
and had to watch much of this going on, at some point you
reach something between 50 and 100% of the hardware limit,
depending on file sizes you copy and who knows what else
overhead you can think of.

Dimiter

======================================================
Dimiter Popoff, TGI             http://www.tgi-sci.com
======================================================
http://www.flickr.com/photos/didi_tgi/

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 07:08 -0700
  Re: Multithreaded disk access Richard Damon <Richard@Damon-Family.org> - 2021-10-15 11:38 -0400
    Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 09:00 -0700
      Re: Multithreaded disk access Richard Damon <Richard@Damon-Family.org> - 2021-10-15 12:48 -0400
        Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 11:40 -0700
  Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-15 18:46 +0300
    Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-15 09:19 -0700
      Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-15 19:28 +0300
  Re: Multithreaded disk access Brett <ggtgp@yahoo.com> - 2021-10-17 20:27 +0000
    Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-17 14:49 -0700
      Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-17 15:01 -0700
      Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-18 01:09 +0300
        Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-18 08:05 -0700
          Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-18 19:25 +0300
            Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-18 13:46 -0700
              Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-19 00:46 +0300
                Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-18 16:17 -0700
                Re: Multithreaded disk access antispam@math.uni.wroc.pl - 2021-10-31 19:21 +0000
                Re: Multithreaded disk access Dimiter_Popoff <dp@tgi-sci.com> - 2021-10-31 21:55 +0200
              Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-25 04:19 -0700
                Re: Multithreaded disk access Richard Damon <Richard@Damon-Family.org> - 2021-10-25 20:23 -0400
                Re: Multithreaded disk access Don Y <blockedofcourse@foo.invalid> - 2021-10-25 22:29 -0700
      Re: Multithreaded disk access Clifford Heath <no.spam@please.net> - 2021-10-18 11:58 +1100

csiph-web