Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90063

Re: Throw the cat among the pigeons

Show all headers | View raw

On Wed, May 6, 2015 at 1:08 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Wednesday 06 May 2015 15:58, Ian Kelly wrote:
>
>> On Tue, May 5, 2015 at 7:27 PM, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> Only the minimum is statistically useful.
>>
>> I disagree. The minimum tells you how fast the code *can* run, under
>> optimal circumstances. The mean tells you how fast it *realistically*
>> runs, under typical load. Both can be useful to measure.
>
> Er, not even close. Running code using timeit is in no way the same as
> running code "for real" under realistic circumstances. The fact that you are
> running the function or code snippet in isolation, in its own scope, via
> exec, rather than as part of some larger script or application, should be a
> hint. timeit itself has overhead, so you cannot measure the time taken by
> the operation alone, you can only measure the time taken by the operation
> within the timeit environment. We have no reason to think that the
> distribution of noise under timeit will be even vaguely similar to the noise
> when running in production.

You also can't be sure that the base time taken by the operation in
your development environment will be comparable to the time taken in
production; different system architectures may produce different
results, and what is faster on your workstation may be slower on a
server.

Also, different algorithms may react to load differently. For example,
an algorithm that goes to different parts of memory frequently may
start thrashing sooner than an algorithm with better spatial locality
if the system is paging a lot. I'll grant that just computing the
means on a workstation that is not under a controlled load is not the
best way to measure this -- but a difference in mean that is not
simply proportional to the difference in min is still potentially
useful information.

> The purpose of timeit is to compare individual algorithms, in as close as
> possible to an equal footing with as little noise as possible. If you want
> to profile code used in a realistic application, use a profiler, not timeit.
> And even that doesn't tell you how fast the code would be alone, because the
> profiler adds overhead.
>
> Besides, "typical load" is a myth -- there is no such thing. A high-end
> Windows web server getting ten thousand hits a minute, a virtual machine
> starved for RAM, a Mac laptop, a Linux server idling away with a load of 0.1
> all day... any of those machines could run your code. How can you *possibly*
> say what is typical? The very idea is absurd.

Agreed.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Throw the cat among the pigeons Cecil Westerhof <Cecil@decebal.nl> - 2015-05-02 16:20 +0200
  Re: Throw the cat among the pigeons Cecil Westerhof <Cecil@decebal.nl> - 2015-05-03 17:12 +0200
    Re: Throw the cat among the pigeons Paul  Moore <p.f.moore@gmail.com> - 2015-05-05 08:47 -0700
      Re: Throw the cat among the pigeons Cecil Westerhof <Cecil@decebal.nl> - 2015-05-05 18:18 +0200
        Re: Throw the cat among the pigeons Dave Angel <davea@davea.name> - 2015-05-05 14:45 -0400
          Re: Throw the cat among the pigeons Cecil Westerhof <Cecil@decebal.nl> - 2015-05-05 21:42 +0200
            Re: Throw the cat among the pigeons Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-06 12:57 +1000
              Re: Throw the cat among the pigeons Chris Angelico <rosuav@gmail.com> - 2015-05-06 14:03 +1000
        Re: Throw the cat among the pigeons Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-05 14:30 -0600
        Re: Throw the cat among the pigeons Terry Reedy <tjreedy@udel.edu> - 2015-05-05 16:46 -0400
          Re: Throw the cat among the pigeons Cecil Westerhof <Cecil@decebal.nl> - 2015-05-05 23:12 +0200
            Re: Throw the cat among the pigeons Terry Reedy <tjreedy@udel.edu> - 2015-05-05 19:22 -0400
        Re: Throw the cat among the pigeons Dave Angel <davea@davea.name> - 2015-05-05 17:00 -0400
        Re: Throw the cat among the pigeons Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-05 15:23 -0600
          Re: Throw the cat among the pigeons Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-06 11:27 +1000
            Re: Throw the cat among the pigeons Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-05 23:58 -0600
              Re: Throw the cat among the pigeons Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-06 17:08 +1000
                Re: Throw the cat among the pigeons Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-06 10:17 -0600
                Re: Throw the cat among the pigeons Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-06 21:23 +0100
        Re: Throw the cat among the pigeons Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-05 15:39 -0600
        Re: Throw the cat among the pigeons Dave Angel <davea@davea.name> - 2015-05-05 17:55 -0400
        Re: Throw the cat among the pigeons Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-06 12:19 +1000
  Re: Throw the cat among the pigeons Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-06 14:05 +1000
    Re: Throw the cat among the pigeons Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-06 16:26 +1000
      Re: Throw the cat among the pigeons Dave Angel <davea@davea.name> - 2015-05-06 09:12 -0400
        Re: Throw the cat among the pigeons Dan Sommers <dan@tombstonezero.net> - 2015-05-07 03:16 +0000
      Re: Throw the cat among the pigeons Chris Angelico <rosuav@gmail.com> - 2015-05-06 23:55 +1000
      Re: Throw the cat among the pigeons Dave Angel <davea@davea.name> - 2015-05-06 10:22 -0400
      Re: Throw the cat among the pigeons Paul Rubin <no.email@nospam.invalid> - 2015-05-06 08:12 -0700
        Re: Throw the cat among the pigeons Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-05-06 17:36 +0200
          Re: Throw the cat among the pigeons Dave Angel <davea@davea.name> - 2015-05-06 16:23 -0400
            Re: Throw the cat among the pigeons Alain Ketterlin <alain@universite-de-strasbourg.fr.invalid> - 2015-05-07 11:14 +0200
              Re: Throw the cat among the pigeons Chris Angelico <rosuav@gmail.com> - 2015-05-07 19:38 +1000
        Re: Throw the cat among the pigeons Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-06 09:47 -0600
          Re: Throw the cat among the pigeons Paul Rubin <no.email@nospam.invalid> - 2015-05-06 10:24 -0700

csiph-web