Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1011059 > unrolled thread

Bug#961370: pv: ETA is wildly off when transfer rate changes

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#961370: pv: ETA is wildly off when transfer rate changes lemonsqueeze <lemonsqueeze@gmx.com> - 2020-05-23 20:00 +0200
    Bug#961370: pv: ETA is wildly off when transfer rate changes Andrew Wood <pv-debianqa@ivarch.com> - 2020-06-28 14:30 +0200
      Bug#961370: pv: ETA is wildly off when transfer rate changes lemonsqueeze <lemonsqueeze@gmx.com> - 2020-07-05 09:40 +0200
    Bug#961370: pv: ETA is wildly off when transfer rate changes lemonsqueeze <lemonsqueeze@gmx.com> - 2020-07-06 18:10 +0200

#1011059 — Bug#961370: pv: ETA is wildly off when transfer rate changes

[Multipart message — attachments visible in raw view] — view raw

Package: pv
Version: 1.6.6-1
Severity: normal
Tags: upstream patch

Hello,

This was reported upstream some time ago but it doesn't seem new pv
versions are coming out so reporting here instead, maybe we can do
something about it in debian.

This was found on ubuntu initially, sorry I haven't tested debian per se
but from what I can see it's affected as well as package sources are
identical.

Consider following scenario:

I was running a really long task that generates a 800Mb file at about
50k/s. Something like:

   while true ; do ls -l /* ; sleep 1 ; done > /tmp/file

Sometime after it started I thought of using pv to keep track of
progress with:

   tail -n +1 -f /tmp/file | pv -s 800m >/dev/null

I was interested in ETA mostly, unfortunately if you do this after a few
mb have been generated the ETA is wildly wrong for a very long time:
transfer rate is really fast initially and fools pv into thinking we're
going to get something like that over time, and it takes a really long
time for the average rate to drop back to 50k/s which is all we're
getting actually.

It'd be nice to have an ETA based on current rate average here, say over
last 30s or last 10s, instead of global rate average.

I've been playing with this patch on top of pv 1.6.6
(also at https://github.com/lemonsqueeze/pv)

Which keeps track of history periodically over a given time window and uses
that to compute current average rate for ETA. Looks like it works pretty
well in this case. Also added a --rate-window option to change time window
(default: last 10s) (name isn't great, maybe could find a better one).


-- System Information:
Debian Release: stretch/sid
  APT prefers xenial-updates
  APT policy: (500, 'xenial-updates'), (500, 'xenial-security'), (500, 'xenial'), (100, 'xenial-backports')
Architecture: i386 (i686)

Kernel: Linux 4.4.0-174-generic (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages pv depends on:
ii  libc6  2.23-0ubuntu11

pv recommends no packages.

Versions of packages pv suggests:
pn  doc-base  <none>

-- no debconf information

[toc] | [next] | [standalone]

#1015554

A new upstream version of PV should be released within the next couple of
months.  The patch kindly provided in this thread was originally sent to me
directly two years ago, so I apologise profusely for the delay.

There have been a couple of other patch submissions around this issue which
I'll need to understand before getting a release together, along with
multiple other TODOs which need organising, but I hope that I have now
sorted out my time management enough that this will not take long.

If anyone can think of a relatively simple and reliable way to test the
behaviour of ETA / rate calculation with bursty input such that it can be
added to "make test", but which doesn't take ages to run, please let me
know.




On Sat, May 23, 2020 at 07:46:20PM +0200, lemonsqueeze wrote:
> Package: pv
> Version: 1.6.6-1
> Severity: normal
> Tags: upstream patch
> 
> Hello,
> 
> This was reported upstream some time ago but it doesn't seem new pv
> versions are coming out so reporting here instead, maybe we can do
> something about it in debian.
> 
> This was found on ubuntu initially, sorry I haven't tested debian per se
> but from what I can see it's affected as well as package sources are
> identical.
> 
> Consider following scenario:
> 
> I was running a really long task that generates a 800Mb file at about
> 50k/s. Something like:
> 
>    while true ; do ls -l /* ; sleep 1 ; done > /tmp/file
> 
> Sometime after it started I thought of using pv to keep track of
> progress with:
> 
>    tail -n +1 -f /tmp/file | pv -s 800m >/dev/null
> 
> I was interested in ETA mostly, unfortunately if you do this after a few
> mb have been generated the ETA is wildly wrong for a very long time:
> transfer rate is really fast initially and fools pv into thinking we're
> going to get something like that over time, and it takes a really long
> time for the average rate to drop back to 50k/s which is all we're
> getting actually.
> 
> It'd be nice to have an ETA based on current rate average here, say over
> last 30s or last 10s, instead of global rate average.
> 
> I've been playing with this patch on top of pv 1.6.6
> (also at https://github.com/lemonsqueeze/pv)
> 
> Which keeps track of history periodically over a given time window and uses
> that to compute current average rate for ETA. Looks like it works pretty
> well in this case. Also added a --rate-window option to change time window
> (default: last 10s) (name isn't great, maybe could find a better one).


-- 
Andrew Wood

[toc] | [prev] | [next] | [standalone]

#1016549

Cool, looking forward to next pv release =)

I was thinking, it seems most command line tools which display ETA
like wget for example exhibit very simple behavior by default:
ETA is displayed every second and average rate used to compute it is
simply average rate during last second (hence wild ETA jumps when
connection speed varies !)

I guess most people would expect that so it probably makes sense for
pv to do this by default also. But it'd certainly be handsome to have
other algorithms to choose from for cases where that is ill-suited.
So the different patches are not necessarily mutually exclusive, could
just add command line options to choose which algorithm to use.

Regarding this patch, name for '--rate-window' option isn't great,
'--eta-window' would be much better. I'll update my patch on github.

Good luck with next pv release !

[toc] | [prev] | [next] | [standalone]

#1016783

updated patch at https://github.com/lemonsqueeze/pv :

- renamed --rate-window option to --eta-window
- fixed crash when rate is zero

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web