Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #81910 > unrolled thread

time.monotonic() roll over

Back to article view | Back to comp.lang.python

Contents

  time.monotonic() roll over "ast" <nomail@invalid.com> - 2014-12-04 16:25 +0100
    Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-04 17:50 +0200
      Re: time.monotonic() roll over Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-04 09:05 -0700
        Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-04 18:21 +0200
          Re: time.monotonic() roll over Chris Angelico <rosuav@gmail.com> - 2014-12-05 03:27 +1100
            Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-04 20:09 +0200
              Re: time.monotonic() roll over Chris Angelico <rosuav@gmail.com> - 2014-12-05 05:29 +1100
                Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-04 20:44 +0200
                  Re: time.monotonic() roll over Chris Angelico <rosuav@gmail.com> - 2014-12-05 07:02 +1100
              Re: time.monotonic() roll over Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-04 12:29 -0700
              Re: time.monotonic() roll over Akira Li <4kir4.1i@gmail.com> - 2014-12-04 23:24 +0300
              Re: time.monotonic() roll over Chris Angelico <rosuav@gmail.com> - 2014-12-05 07:33 +1100
                Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-04 22:51 +0200
              Re: time.monotonic() roll over Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-04 15:17 -0700
                Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-05 00:44 +0200
                  Re: time.monotonic() roll over Ian Kelly <ian.g.kelly@gmail.com> - 2014-12-05 10:04 -0700
                    Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-05 20:48 +0200
              Re: time.monotonic() roll over Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-05 11:39 +1100
                Re: time.monotonic() roll over Dave Angel <davea@davea.name> - 2014-12-04 20:21 -0500
                  Re: time.monotonic() roll over Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-05 13:54 +1100
                    Re: time.monotonic() roll over Dave Angel <davea@davea.name> - 2014-12-05 00:29 -0500
                    Re: time.monotonic() roll over Lele Gaifax <lele@metapensiero.it> - 2014-12-05 08:56 +0100
                      Re: time.monotonic() roll over Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-05 21:09 +1100
                    Re: time.monotonic() roll over Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-12-05 12:12 -0500
                Re: time.monotonic() roll over Marko Rauhamaa <marko@pacujo.net> - 2014-12-05 08:17 +0200
                  Re: time.monotonic() roll over Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-12-05 12:31 -0500
                Re: time.monotonic() roll over Akira Li <4kir4.1i@gmail.com> - 2014-12-05 12:48 +0300
      Re: time.monotonic() roll over Chris Angelico <rosuav@gmail.com> - 2014-12-05 03:12 +1100
      Re: time.monotonic() roll over random832@fastmail.us - 2014-12-04 11:51 -0500
    Re: time.monotonic() roll over Nobody <nobody@nowhere.invalid> - 2014-12-05 00:54 +0000

Page 1 of 2  [1] 2  Next page →

#81910 — time.monotonic() roll over

Hello,

Does any body know when time.monotonic() rolls over ?

On python doc https://docs.python.org/3/library/time.html
it is said  every 49.7 days on Windows versions older than 
Vista. For more recent Windows, it is sais that  monotonic() 
is system-wide but they dont say anything about roll over,
probably it hasn't changed.

Anyway, if someone need a monotonic clock, he may think
to use either time.time() or time.monotonic().
There is no roll over problem with time.time() since the very
first one in planned far in the future, but time.time() can go
backward when a date update throught NTP server is done.

time.monotonic() is monotonic but roll over often (every 49.7 
days)

So what to do ? 

Using time.monotonic() and take some actions if a roll over 
is detected ?

[toc] | [next] | [standalone]

#81913

"ast" <nomail@invalid.com>:

> Does any body know when time.monotonic() rolls over ?

Never, according to the documentation you linked.

Admittedly, the documentation confuses the reader by chatting about some
irrelevant internal Windows details.

Also, the tone of the documentation raises a suspicion that this code
might return a bad value on Windows:

    def fifty_days():
        a = time.monotonic()
        time.sleep(50 * 24 * 3600)
        return time.monotonic() - a

That is, the internal integer wrap is not guarded against between the
calls to time.monotonic(), maybe.


Marko

[toc] | [prev] | [next] | [standalone]

#81915

[Multipart message — attachments visible in raw view] — view raw

On Dec 4, 2014 8:56 AM, "Marko Rauhamaa" <marko@pacujo.net> wrote:
>
> "ast" <nomail@invalid.com>:
>
> > Does any body know when time.monotonic() rolls over ?
>
> Never, according to the documentation you linked.
>
> Admittedly, the documentation confuses the reader by chatting about some
> irrelevant internal Windows details.

Not entirely irrelevant. The implication is that if you go more than 49
days without calling the function on old Windows systems, rollovers could
be missed, which is good to know about. The result would still be
monotonic, but it wouldn't accurately reflect the time elapsed.

[toc] | [prev] | [next] | [standalone]

#81917

Ian Kelly <ian.g.kelly@gmail.com>:

> The implication is that if you go more than 49 days without calling
> the function on old Windows systems, rollovers could be missed, which
> is good to know about.

The implication is all but clear, but that was my suspicion. It would be
so bad that the documentation should be much more explicit about it.

No, that would be a Python standard library bug, period.


Marko

[toc] | [prev] | [next] | [standalone]

#81918

On Fri, Dec 5, 2014 at 3:21 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Ian Kelly <ian.g.kelly@gmail.com>:
>
>> The implication is that if you go more than 49 days without calling
>> the function on old Windows systems, rollovers could be missed, which
>> is good to know about.
>
> The implication is all but clear, but that was my suspicion. It would be
> so bad that the documentation should be much more explicit about it.
>
> No, that would be a Python standard library bug, period.

It's not a Python issue. Python can't do anything more than ask the
system, and if the system's value rolls over several times a year,
Python can't magically cure that. The information has already been
lost.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81928

Chris Angelico <rosuav@gmail.com>:

> It's not a Python issue. Python can't do anything more than ask the
> system, and if the system's value rolls over several times a year,
> Python can't magically cure that. The information has already been
> lost.

Sure it could by having an invisible background thread occasionally call
time.monotonic(). It could even be done on the side without a thread.

Anyway, the idea of a clock is complicated:

 * the program could be stopped by a STOP signal

 * the program could be suspended from power management

 * the program could be resurrected from a virtual machine snapshot

 * the program could be migrated from a different physical machine

So, if I call

   time.sleep(86400)

and the program is suspended for 24 hours, should time.sleep() return
right after it is resumed or after another 24 hours?


Marko

[toc] | [prev] | [next] | [standalone]

#81931

On Fri, Dec 5, 2014 at 5:09 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Sure it could by having an invisible background thread occasionally call
> time.monotonic(). It could even be done on the side without a thread.

No, it can't be solved by anything in that process, because...

>  * the program could be stopped by a STOP signal

... what you said  There's no way to guarantee to keep calling the
function. You have to depend on upstream.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81934

Chris Angelico <rosuav@gmail.com>:

> ... what you said  There's no way to guarantee to keep calling the
> function. You have to depend on upstream.

The caveats I listed are real concerns for the modern-day programmer.
However, they are of a different nature than the complaint against
time.monotonic().


Marko

[toc] | [prev] | [next] | [standalone]

#81940

On Fri, Dec 5, 2014 at 5:44 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> ... what you said  There's no way to guarantee to keep calling the
>> function. You have to depend on upstream.
>
> The caveats I listed are real concerns for the modern-day programmer.
> However, they are of a different nature than the complaint against
> time.monotonic().

But they're also reasons you can't have Python code around the
problem. Anyway, it's only an issue for one platform, and only for
some versions of it; basically, if you're targeting anything other
than Windows XP, you should be able to use time.monotonic() without
concerns.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81936

[Multipart message — attachments visible in raw view] — view raw

On Thu, Dec 4, 2014 at 11:09 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>
> Chris Angelico <rosuav@gmail.com>:
>
> > It's not a Python issue. Python can't do anything more than ask the
> > system, and if the system's value rolls over several times a year,
> > Python can't magically cure that. The information has already been
> > lost.
>
> Sure it could by having an invisible background thread occasionally call
> time.monotonic(). It could even be done on the side without a thread.
>
> Anyway, the idea of a clock is complicated:
>
>  * the program could be stopped by a STOP signal
>
>  * the program could be suspended from power management
>
>  * the program could be resurrected from a virtual machine snapshot
>
>  * the program could be migrated from a different physical machine

This seems like a lot of effort to unreliably design around a problem that
will matter to only a tiny fraction of users.

[toc] | [prev] | [next] | [standalone]

#81945

Ian Kelly <ian.g.kelly@gmail.com> writes:

> On Thu, Dec 4, 2014 at 11:09 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>>
>> Chris Angelico <rosuav@gmail.com>:
>>
>> > It's not a Python issue. Python can't do anything more than ask the
>> > system, and if the system's value rolls over several times a year,
>> > Python can't magically cure that. The information has already been
>> > lost.
>>
>> Sure it could by having an invisible background thread occasionally call
>> time.monotonic(). It could even be done on the side without a thread.
>>
>> Anyway, the idea of a clock is complicated:
>>
>>  * the program could be stopped by a STOP signal
>>
>>  * the program could be suspended from power management
>>
>>  * the program could be resurrected from a virtual machine snapshot
>>
>>  * the program could be migrated from a different physical machine
>
> This seems like a lot of effort to unreliably design around a problem that
> will matter to only a tiny fraction of users.

- people's computers are mostly on batteries (laptops, tablets,
  smartphones) -- "suspended from power management" use case
- corporations's computations are mostly virtualized -- possible
  "ressurected", "migrated" use case

i.e., the opposite might be true -- non-virtualized PCs connected to AC
are (becoming) minority.


--
Akira

[toc] | [prev] | [next] | [standalone]

#81947

On Fri, Dec 5, 2014 at 7:24 AM, Akira Li <4kir4.1i@gmail.com> wrote:
>> This seems like a lot of effort to unreliably design around a problem that
>> will matter to only a tiny fraction of users.
>
> - people's computers are mostly on batteries (laptops, tablets,
>   smartphones) -- "suspended from power management" use case
> - corporations's computations are mostly virtualized -- possible
>   "ressurected", "migrated" use case
>
> i.e., the opposite might be true -- non-virtualized PCs connected to AC
> are (becoming) minority.

That's a massive over-simplification, of course, but given that my
current desktop computer is running a dozen or so VMs for various
purposes (usually not more than 3-4 concurrently), I can't disagree
with you. However, there still are plenty of computers that are always
either fully running, or fully shut down; just because people _can_
suspend with applications running doesn't mean they _will_. (Quite a
few of my VMs, for instance, do not get saved/suspended - I shut them
down whenever I'm done with them. Even when I do suspend a VM, I often
terminate applications in it, and just use suspension to save having
to boot the OS every time. But that's partly because those VMs are the
ones running Windows for specific proprietary apps, and thus are
coping with the vagaries of those apps.)

In any case, it's not at all a problem to have the protection on
systems that won't actually need it. Much better than lacking the
protection on a system that does.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81949

Chris Angelico <rosuav@gmail.com>:

> Even when I do suspend a VM, I often terminate applications in it, and
> just use suspension to save having to boot the OS every time.

One interesting detail is DHCP leases.

When it is resumed from suspension, a linux computer thinks it still has
the old IP address, which might have already been granted to some other
computer on the network. Thus, linux distros are equipped with hacky
scripts that get executed soon after the machine is resumed (and before
a lot of damage is done). One of them forces a DHCP renegotiation.

A similar scheme is missing from virtual machines: a VM that wakes up
from a snapshot doesn't automatically go through the resume scripts,
even though it arguably should.

Systemd probably shuffles the situation still more.


Marko

[toc] | [prev] | [next] | [standalone]

#81953

[Multipart message — attachments visible in raw view] — view raw

On Thu, Dec 4, 2014 at 1:24 PM, Akira Li <4kir4.1i@gmail.com> wrote:
>
> Ian Kelly <ian.g.kelly@gmail.com> writes:
> > This seems like a lot of effort to unreliably design around a problem
that
> > will matter to only a tiny fraction of users.
>
> - people's computers are mostly on batteries (laptops, tablets,
>   smartphones) -- "suspended from power management" use case
> - corporations's computations are mostly virtualized -- possible
>   "ressurected", "migrated" use case
>
> i.e., the opposite might be true -- non-virtualized PCs connected to AC
> are (becoming) minority.

By "tiny fraction of users" I was referring to people who a) work with
Python 3.3+, b) use time.monotonic in their code, and c) possibly have said
code running on versions of Windows older than Vista; not people who
suspend their systems or use VMs.

It's not clear to me whether those cases are relevant to the rollover
concern anyway. I wouldn't be shocked if the GetTickCount() function simply
stopped increasing while the system is suspended, since after all it's not
"ticking" during that time.

[toc] | [prev] | [next] | [standalone]

#81955

Ian Kelly <ian.g.kelly@gmail.com>:

> It's not clear to me whether those cases are relevant to the rollover
> concern anyway. I wouldn't be shocked if the GetTickCount() function
> simply stopped increasing while the system is suspended, since after
> all it's not "ticking" during that time.

So, what's the semantics of time.sleep(), select.select() et al wrt
process or machine suspension?


Marko

[toc] | [prev] | [next] | [standalone]

#82006

[Multipart message — attachments visible in raw view] — view raw

On Thu, Dec 4, 2014 at 3:44 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>
> Ian Kelly <ian.g.kelly@gmail.com>:
>
> > It's not clear to me whether those cases are relevant to the rollover
> > concern anyway. I wouldn't be shocked if the GetTickCount() function
> > simply stopped increasing while the system is suspended, since after
> > all it's not "ticking" during that time.
>
> So, what's the semantics of time.sleep(), select.select() et al wrt
> process or machine suspension?

Rather than continue to speculate, I just tested this. I ran the following
script in a Windows 7 VM and paused the VM for part of the duration of the
sleep call. I unpaused well before the sleep call would normally have ended.

>>> import time
>>> def test():
...   t1 = time.time()
...   m1 = time.monotonic()
...   time.sleep(60)
...   t2 = time.time()
...   m2 = time.monotonic()
...   return t2 - t1, m2 - m1
...
>>> test()
(84.92642459869386, 60.002399999999994)

As you can see, the sleep call and the monotonic time elapsed both appear
to exclude the period when the VM was paused.

Then I tried the same with a MacBook and putting the system to sleep
instead of pausing a VM:

>>> test()
(59.889225006103516, 60.00102640298428)

Reducing the sleep time to 10 seconds and suspending the process with
Ctrl-Z produces a similar result on the MacBook:
(10.000508069992065, 10.000266332994215)

But the story is different in Linux. Here's what I got with the same Ctrl-Z
test:
(16.000478506088257, 16.000478034024127)

I think the relative roundness of those numbers is just coincidence. I
tried a second time and got:
(12.613622903823853, 12.613622067990946)

So it seems that it can go either way depending on the OS and perhaps the
mechanism of suspension.

[toc] | [prev] | [next] | [standalone]

#82015

Ian Kelly <ian.g.kelly@gmail.com>:

> On Thu, Dec 4, 2014 at 3:44 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> So, what's the semantics of time.sleep(), select.select() et al wrt
>> process or machine suspension?
>
> Rather than continue to speculate, I just tested this.

The test, of course, can only have one useful result...

> So it seems that it can go either way depending on the OS and perhaps the
> mechanism of suspension.

... and you got it.

If the semantics had coincided, the question would remain unresolved; it
could just have been a coincidence.

However, the variance in the results means the semantics are
unspecified.

I think programmer's would need facilities to determine and control the
suspension behavior.

All things being equal, *early* wakeup is preferable to late wakeup
because the application can then check for a spurious wakeup and go back
to sleep. The reverse situation would be painful to deal with.


Marko

[toc] | [prev] | [next] | [standalone]

#81965

Marko Rauhamaa wrote:

> So, if I call
> 
> time.sleep(86400)
> 
> and the program is suspended for 24 hours, should time.sleep() return
> right after it is resumed or after another 24 hours?

If the program is suspended, then no time should pass for that program.
Since sleep() is given in terms of a duration, not an absolute time ("sleep
until now + 24 hours"), if no time passes for that program, sleep() should
sleep for 24 hours *after the program resumes*.

Unfortunately a lot of systems get that wrong. E.g. I just ran "sleep 30"
from my Linux shell, immediately paused it using Ctrl-Z, waited a couple of
minutes, and used fg to continue. It returned immediately.

Why is this behaviour wrong?

Consider why people might call sleep. They're normally calling it to wait
for something else to complete. Often that something else is an external
process ("wait a few seconds to let the hard drive finish syncing, wait for
the web server to get less busy, wait for the user to catch up...") but it
might be another thread in the same process. If the process is suspended,
that other thread won't get to run, and so even though time on the outside
has continued to move forward, from the perspective of the process, it
hasn't.

The same applies even more so if the process is running in a virtual machine
which has just been suspended.

The same applies if the system clock is adjusted mid-sleep. If I call sleep
60, then immediately adjust the clock forward an hour (say, due to a
daylight savings adjustment), that shouldn't cause sleep to return. It
should still suspend for 60 seconds.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81967

On 12/04/2014 07:39 PM, Steven D'Aprano wrote:
> Marko Rauhamaa wrote:
>
>> So, if I call
>>
>> time.sleep(86400)
>>
>> and the program is suspended for 24 hours, should time.sleep() return
>> right after it is resumed or after another 24 hours?
>
> If the program is suspended, then no time should pass for that program.
> Since sleep() is given in terms of a duration, not an absolute time ("sleep
> until now + 24 hours"), if no time passes for that program, sleep() should
> sleep for 24 hours *after the program resumes*.
>
> Unfortunately a lot of systems get that wrong. E.g. I just ran "sleep 30"
> from my Linux shell, immediately paused it using Ctrl-Z, waited a couple of
> minutes, and used fg to continue. It returned immediately.
>
> Why is this behaviour wrong?
>
> Consider why people might call sleep. They're normally calling it to wait
> for something else to complete. Often that something else is an external
> process ("wait a few seconds to let the hard drive finish syncing, wait for
> the web server to get less busy, wait for the user to catch up...") but it
> might be another thread in the same process. If the process is suspended,
> that other thread won't get to run, and so even though time on the outside
> has continued to move forward, from the perspective of the process, it
> hasn't.
>
> The same applies even more so if the process is running in a virtual machine
> which has just been suspended.
>
> The same applies if the system clock is adjusted mid-sleep. If I call sleep
> 60, then immediately adjust the clock forward an hour (say, due to a
> daylight savings adjustment), that shouldn't cause sleep to return. It
> should still suspend for 60 seconds.

And I say you have it exactly backwards.  People should NOT call sleep() 
to measure something, they should only call it to release the processor 
for "up to" a specified time.  That time should normally be a major 
fraction of the time that some external event is expected to take.  Then 
the code should check the external event and sleep some more if necessary.

If efficiency were not an issue, the code would just do a busy loop, 
checking the external condition repeatedly.

Since the OS has no way of knowing whether the thing being waited for is 
a thread, another process, a human being, a network operation, or the 
end of the world, the interpretation of sleep needs to be the most 
conservative one.  There are many ways of suspending a process, and some 
of them will also suspend the external event.  Since the OS cannot know 
which case is significant, it has to return control to the caller at the 
soonest of the many possible interpretations.


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#81971

Dave Angel wrote:

> On 12/04/2014 07:39 PM, Steven D'Aprano wrote:
>> Marko Rauhamaa wrote:
>>
>>> So, if I call
>>>
>>> time.sleep(86400)
>>>
>>> and the program is suspended for 24 hours, should time.sleep() return
>>> right after it is resumed or after another 24 hours?
>>
>> If the program is suspended, then no time should pass for that program.
>> Since sleep() is given in terms of a duration, not an absolute time
>> ("sleep until now + 24 hours"), if no time passes for that program,
>> sleep() should sleep for 24 hours *after the program resumes*.
>>
>> Unfortunately a lot of systems get that wrong. E.g. I just ran "sleep 30"
>> from my Linux shell, immediately paused it using Ctrl-Z, waited a couple
>> of minutes, and used fg to continue. It returned immediately.
>>
>> Why is this behaviour wrong?
>>
>> Consider why people might call sleep. They're normally calling it to wait
>> for something else to complete. Often that something else is an external
>> process ("wait a few seconds to let the hard drive finish syncing, wait
>> for the web server to get less busy, wait for the user to catch up...")
>> but it might be another thread in the same process. If the process is
>> suspended, that other thread won't get to run, and so even though time on
>> the outside has continued to move forward, from the perspective of the
>> process, it hasn't.
>>
>> The same applies even more so if the process is running in a virtual
>> machine which has just been suspended.
>>
>> The same applies if the system clock is adjusted mid-sleep. If I call
>> sleep 60, then immediately adjust the clock forward an hour (say, due to
>> a daylight savings adjustment), that shouldn't cause sleep to return. It
>> should still suspend for 60 seconds.
> 
> And I say you have it exactly backwards.  People should NOT call sleep()
> to measure something, 

I never suggested you call sleep to measure something. You call sleep to
*wait* for something. I even used the word "wait" four times in the above.


> they should only call it to release the processor 
> for "up to" a specified time.

That's silly. Zero is "up to" any positive value you want. Do you really
intend to say that this would be an acceptable implementation of sleep()?

def sleep(delay):
    return



> That time should normally be a major 
> fraction of the time that some external event is expected to take.  Then
> the code should check the external event and sleep some more if necessary.
> 
> If efficiency were not an issue, the code would just do a busy loop,
> checking the external condition repeatedly.

In many cases, you don't have an external condition that can be easily or
practically checked. So you estimate how long you need to wait, add a
safety margin, and write "sleep 5" (for some value of 5), and hope.

If you are a sys admin writing a script that runs during boot time, you do
that a lot. Or at least the ones I work with do :-)


> Since the OS has no way of knowing whether the thing being waited for is
> a thread, another process, a human being, a network operation, or the
> end of the world, the interpretation of sleep needs to be the most
> conservative one.

I agree! And returning immediately is not the most conservative one.

The most conservative approach is to assume that while you're suspended,
*everything else* is suspended too, so when you resume you still have to
sleep for the full N seconds.


> There are many ways of suspending a process, and some 
> of them will also suspend the external event.  Since the OS cannot know
> which case is significant, it has to return control to the caller at the
> soonest of the many possible interpretations.

/s/latest/



-- 
Steven

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web