Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #30332 > unrolled thread

Reducing cache/buffer for faster display

Back to article view | Back to comp.lang.python

Contents

  Reducing cache/buffer for faster display Rikishi42 <skunkworks@rikishi42.net> - 2012-09-27 23:57 +0200
    Re: Reducing cache/buffer for faster display Chris Angelico <rosuav@gmail.com> - 2012-09-28 08:09 +1000
      Re: Reducing cache/buffer for faster display John Gordon <gordon@panix.com> - 2012-09-27 22:25 +0000
        Re: Reducing cache/buffer for faster display Chris Angelico <rosuav@gmail.com> - 2012-09-28 08:38 +1000
          Re: Reducing cache/buffer for faster display Rikishi42 <skunkworks@rikishi42.net> - 2012-09-28 02:05 +0200
            Re: Reducing cache/buffer for faster display Chris Angelico <rosuav@gmail.com> - 2012-09-28 18:08 +1000
              Re: Reducing cache/buffer for faster display Rikishi42 <skunkworks@rikishi42.net> - 2012-09-29 02:31 +0200
        Re: Reducing cache/buffer for faster display Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-09-27 21:46 -0400
          Re: Reducing cache/buffer for faster display Rikishi42 <skunkworks@rikishi42.net> - 2012-09-29 02:20 +0200
            Re: Reducing cache/buffer for faster display Hans Mulder <hansmu@xs4all.nl> - 2012-09-29 14:32 +0200
            Re: Reducing cache/buffer for faster display Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-09-29 14:12 -0400
      Re: Reducing cache/buffer for faster display Rikishi42 <skunkworks@rikishi42.net> - 2012-09-28 02:15 +0200

#30332 — Reducing cache/buffer for faster display

I have these 2 scripts that are very heavy on the file i/o, consume a very
reasonable amount of cpu and output their counters at a - very - relaxed
pace to the console. The output is very simply done using something like:

   print "files:", nFiles, "\r",


Yet alltough there is no real reason for it, even a pace of a print every
10-30 secs will be cached, only to actually show an output update every 1-2
min or so.

When I run the scripts with "python -u myscript.py", the output is complete,
very speedy and without any kind of impact on the actual work being one.


Can this option be called from within the script? Or is there another option
to make the display "a bit" speedier ?


Runnning Python 2.7.3, but it seems to me I've allready had this problem a
long, long time ago with other releases.



-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [next] | [standalone]

#30335

On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <skunkworks@rikishi42.net> wrote:
> I have these 2 scripts that are very heavy on the file i/o, consume a very
> reasonable amount of cpu and output their counters at a - very - relaxed
> pace to the console. The output is very simply done using something like:
>
>    print "files:", nFiles, "\r",
>
>
> Yet alltough there is no real reason for it, even a pace of a print every
> 10-30 secs will be cached, only to actually show an output update every 1-2
> min or so.

Yup! Just add a call to sys.stdout.flush() after each print.

ChrisA

[toc] | [prev] | [next] | [standalone]

#30336

In <mailman.1522.1348783807.27098.python-list@python.org> Chris Angelico <rosuav@gmail.com> writes:

> On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <skunkworks@rikishi42.net> wrote:
> > I have these 2 scripts that are very heavy on the file i/o, consume a very
> > reasonable amount of cpu and output their counters at a - very - relaxed
> > pace to the console. The output is very simply done using something like:
> >
> >    print "files:", nFiles, "\r",
> >
> >
> > Yet alltough there is no real reason for it, even a pace of a print every
> > 10-30 secs will be cached, only to actually show an output update every 1-2
> > min or so.

> Yup! Just add a call to sys.stdout.flush() after each print.

Isn't terminal output line-buffered?  I don't understand why there would
be an output delay.  (Unless the "\r" is messing things up...)

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon@panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"

[toc] | [prev] | [next] | [standalone]

#30337

On Fri, Sep 28, 2012 at 8:25 AM, John Gordon <gordon@panix.com> wrote:
> Isn't terminal output line-buffered?  I don't understand why there would
> be an output delay.  (Unless the "\r" is messing things up...)

This is a classic progress-indication case, which does indeed mess up
line-buffering. The carriage return (and no line feed, done in the
Python 2 style of a trailing comma) puts the cursor back to the
beginning of the line, ready to overwrite, and ripe for one of those
old favorite incomplete overwrite errors - if nFiles monotonically
increases, it's fine, but if it decreases, the display can get ugly.

ChrisA

[toc] | [prev] | [next] | [standalone]

#30343

On 2012-09-27, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Sep 28, 2012 at 8:25 AM, John Gordon <gordon@panix.com> wrote:
>> Isn't terminal output line-buffered?  I don't understand why there would
>> be an output delay.  (Unless the "\r" is messing things up...)
>
> This is a classic progress-indication case, which does indeed mess up
> line-buffering. The carriage return (and no line feed, done in the
> Python 2 style of a trailing comma) puts the cursor back to the
> beginning of the line, ready to overwrite, and ripe for one of those
> old favorite incomplete overwrite errors - if nFiles monotonically
> increases, it's fine, but if it decreases, the display can get ugly.

True, but that wasn't the problem here. The updates where. Thanks for the
given answer, I'll try it.

The scripts in question only increase numbers. But should that not be the
case, solutions are simple enough. The numbers can be formatted to have a
fixed size. In the case of random line contents (a list of filesnames, say)
it's enough to create an output function that is aware of the length of the
previously printed line, and add enough spaces to the current one to wipe
exess content.


Thanks again for the suggestion.


-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [prev] | [next] | [standalone]

#30367

On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 <skunkworks@rikishi42.net> wrote:
> The scripts in question only increase numbers. But should that not be the
> case, solutions are simple enough. The numbers can be formatted to have a
> fixed size. In the case of random line contents (a list of filesnames, say)
> it's enough to create an output function that is aware of the length of the
> previously printed line, and add enough spaces to the current one to wipe
> exess content.

Yep, that's a pretty effective way to do it. One simple method to it
is to format the whole string as a single whole, then left justify it
in a field of (say) 79 characters, and output that:

msg = "Progress: %d%% (%d/%d)... %s" % (done*100/total, done, total,
current_file)
print msg.ljust(79)+"\r",
sys.stdout.flush()

ChrisA

[toc] | [prev] | [next] | [standalone]

#30447

On 2012-09-28, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 <skunkworks@rikishi42.net> wrote:
>> The scripts in question only increase numbers. But should that not be the
>> case, solutions are simple enough. The numbers can be formatted to have a
>> fixed size. In the case of random line contents (a list of filesnames, say)
>> it's enough to create an output function that is aware of the length of the
>> previously printed line, and add enough spaces to the current one to wipe
>> exess content.
>
> Yep, that's a pretty effective way to do it. One simple method to it
> is to format the whole string as a single whole, then left justify it
> in a field of (say) 79 characters, and output that:
>
> msg = "Progress: %d%% (%d/%d)... %s" % (done*100/total, done, total,
> current_file)
> print msg.ljust(79)+"\r",
> sys.stdout.flush()

Mmm, I allmost went for that. It's elegant, simple and clear. But there's
one drawback: I usually reduce the terminal's window to take up less desktop
surface during those long runs.  
So fixing it to 79 chars won't do.  And I'm not even tempted to go for a
detection of the width of the terminal from within the script.  The idea is
after all to keep the scripts simple (syntax) and light (execution).

Well, good night everyone.

-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [prev] | [next] | [standalone]

#30346

On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <gordon@panix.com>
declaimed the following in gmane.comp.python.general:

> 
> Isn't terminal output line-buffered?  I don't understand why there would
> be an output delay.  (Unless the "\r" is messing things up...)

	It's the trailing ,		The \r is being used to reset to the
beginning of the console line, but the comma "says" more output for
/this/ line will be coming... So no output until explicitly flushed, or
a new-line is issued.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#30446

On 2012-09-28, Dennis Lee Bieber <wlfraed@ix.netcom.com> wrote:
> On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <gordon@panix.com>
> declaimed the following in gmane.comp.python.general:
>
>> 
>> Isn't terminal output line-buffered?  I don't understand why there would
>> be an output delay.  (Unless the "\r" is messing things up...)
>
> 	It's the trailing ,		The \r is being used to reset to the
> beginning of the console line, but the comma "says" more output for
> /this/ line will be coming... So no output until explicitly flushed, or
> a new-line is issued.

Well, the \r seems to be the problem, allright.
But output was not completely blocked, just delayed a very long time.  

So perhaps flushing and a sending a newline aren't the only triggers for
output.  Perhaps there's a maximum delay or a maximum cumulated size, and
the output is flushed when such a limit is reached.

Anyway, that's mainly academic. I doubt there will be a correction to
that behaviour. 

-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [prev] | [next] | [standalone]

#30478

On 29/09/12 02:20:50, Rikishi42 wrote:
> On 2012-09-28, Dennis Lee Bieber <wlfraed@ix.netcom.com> wrote:
>> On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <gordon@panix.com>
>> declaimed the following in gmane.comp.python.general:
>>
>>>
>>> Isn't terminal output line-buffered?  I don't understand why there would
>>> be an output delay.  (Unless the "\r" is messing things up...)
>>
>> 	It's the trailing ,		The \r is being used to reset to the
>> beginning of the console line, but the comma "says" more output for
>> /this/ line will be coming... So no output until explicitly flushed, or
>> a new-line is issued.
> 
> Well, the \r seems to be the problem, allright.
> But output was not completely blocked, just delayed a very long time.  
> 
> So perhaps flushing and a sending a newline aren't the only triggers for
> output.  Perhaps there's a maximum delay or a maximum cumulated size, and
> the output is flushed when such a limit is reached.

There's a maximum cumulated size; it's called the buffer size.
Output goes into a buffer, and when the buffer is full, it's
printed all at once.

One way to avoid it, is to use an unbuffered stream.

Another, more efficient, way to avoid it, is to invoke the
stream's .flush() method after writing to it.

> Anyway, that's mainly academic. I doubt there will be a correction to
> that behaviour. 

It's an optimization.  When it was invented, 40 years ago, it was a
really necessary to do this, to get something resembling performance.

The performance of a system without stream buffering would probably
be tolerable on modern hardware.  But the people maintaining Python
are unlikely to cut out buffering, because few people would benefit
(yours is pretty much the only use case where buffereing hurts) and
some would suffer (those who write many short strings to a disk file).


Hope this helps,

-- HansM

[toc] | [prev] | [next] | [standalone]

#30519

On Sat, 29 Sep 2012 02:20:50 +0200, Rikishi42 <skunkworks@rikishi42.net>
declaimed the following in gmane.comp.python.general:


> 
> Well, the \r seems to be the problem, allright.
> But output was not completely blocked, just delayed a very long time.  
>
	Of course -- once the output buffer was filled, the system would
output the data...
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#30344

On 2012-09-27, Chris Angelico <rosuav@gmail.com> wrote:
> On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <skunkworks@rikishi42.net> wrote:
>> I have these 2 scripts that are very heavy on the file i/o, consume a very
>> reasonable amount of cpu and output their counters at a - very - relaxed
>> pace to the console. The output is very simply done using something like:
>>
>>    print "files:", nFiles, "\r",
>>
>>
>> Yet alltough there is no real reason for it, even a pace of a print every
>> 10-30 secs will be cached, only to actually show an output update every 1-2
>> min or so.
>
> Yup! Just add a call to sys.stdout.flush() after each print.

Update: tried it, ran it, I love it.

Thanks !


-- 
When in doubt, use brute force.
                -- Ken Thompson

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web