Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch > #2186

Re: Memory Transactions per Second.

Show all headers | View raw

And Skybuck Flying writes:
> I would like to know how many "random locations" in a memory
> system can be read or written to per second.

And Andy Glew writes:
> This is a really important metric.  But, unfortunately, it is
> often hard to tell from published specs or benchmarks.

A very rough but surprisingly useful upper bound with which I've
been working: Consider the tightest bottleneck between memory
request issuers (CPUs) and servicers (memory) and adjust by the
request concurrency.  It's a bit obvious, but it seems to work
fairly well.

Let R be the rate of memory requests being issued.  Let T be the
density of memory requests per CPU clock cycle (assume global
sync for modeling) to relate it back to the executed code and
number of processors.  Let's say that the bottleneck for issuing
memory operations is the interconnect between CPU and memory.  If
that interconnect issues operations at 220 Mop/s per processor,
that gives you an issuing slope of R_issue = 0.22*T Gop/s.  In
real life, that rate needs some split between reads and writes.

Let's further say the memory's running at 100MHz, giving a peak
service rate of 100 Mop/s per memory unit.  If you have M memory
units, the magic ideal peak service rate is R_ideal = 0.1*M Gop/s
*if* you can keep enough requests in flight.  Let F be the
fraction of latency you can tolerate (number of requests in
flight over number needed to cover latency).  The R_mem = 0.1*M*F
Gop/s.  Yes, this is an overestimate, but surprisingly it doesn't
seem too horrible.

So a given code should be bound by the least of R_issue and
R_mem.  This forms a nice roofline model.  Seems to work
beautifully for predicting when speed-up tops out for simple
graph codes on a Cray XMT (using above specs with twiddles for
bisection bandwidth, need to edit and re-submit paper on it,
damned hard to fit into ten pages).

Still poking at the right bottlenecks to measure on QPI-based
servers (97 outstanding ops, iirc) and the Intel SCC (a ton of
modeling complexity), but the rough shape looks right in
microbenchmarks.  And for these, a request is a cache line and
not a 64-bit work.  Doesn't matter much for a microbenchmark but
does for a graph kernel.

Haven't thought hard about GPGPUs yet, but they're on my to-do
list.  I don't wanna touch graph algorithms there, but some of
our students do, so good. ;)

> As Mitch says, the only really good way to do this is to run
> your own benchmark.

A good one (for the peanut gallery, as many of us have written
this a zillion times) is to generate a long array of 0...BIG,
permute it randomly, and have each thread start walking it
somewhere for a large number of iterations.  Time the walking
part.  You can adjust the requested concurrency by having each
thread start at a few positions and follow each of them in
lock-step.  And then you can poke at the read/write split by
having the threads copy the data into a second array.
-- 
Jason

Back to comp.arch | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-12 07:57 +0200
  Re: Memory Transactions per Second. MitchAlsup <MitchAlsup@aol.com> - 2011-06-12 08:59 -0700
    Re: Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-12 23:42 +0200
      Re: Memory Transactions per Second. Robert Redelmeier <redelm@ev1.net.invalid> - 2011-06-13 02:49 +0000
        Re: Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-13 16:50 +0200
          Re: Memory Transactions per Second. Robert Redelmeier <redelm@ev1.net.invalid> - 2011-06-15 02:49 +0000
            Re: Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-15 06:55 +0200
  Re: Memory Transactions per Second. "Andy \"Krazy\" Glew" <andy@SPAM.comp-arch.net> - 2011-06-12 12:51 -0700
    Re: Memory Transactions per Second. MitchAlsup <MitchAlsup@aol.com> - 2011-06-12 14:40 -0700
    Re: Memory Transactions per Second. Jason Riedy <jason@lovesgoodfood.com> - 2011-06-13 09:33 -0400
  Re: Memory Transactions per Second. mr computer <microbrainz@gmail.com> - 2011-06-12 16:06 -0700
    Re: Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-13 16:29 +0200
      Re: Memory Transactions per Second. mr computer <microbrainz@gmail.com> - 2011-06-13 07:54 -0700
        Re: Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-13 17:08 +0200
          Re: Memory Transactions per Second. MitchAlsup <MitchAlsup@aol.com> - 2011-06-13 11:50 -0700
            Re: Memory Transactions per Second. "Skybuck Flying" <Windows7IsOK@DreamPC2006.com> - 2011-06-14 04:07 +0200

csiph-web