Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #104553 > unrolled thread

Other difference with Perl: Python scripts in a pipe

Back to article view | Back to comp.lang.python

Contents

  Other difference with Perl: Python scripts in a pipe Fillmore <fillmore_remove@hotmail.com> - 2016-03-10 16:33 -0500
    Re: Other difference with Perl: Python scripts in a pipe Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-10 14:46 -0700
      Re: Other difference with Perl: Python scripts in a pipe Fillmore <fillmore_remove@hotmail.com> - 2016-03-10 17:04 -0500
    Re: Other difference with Perl: Python scripts in a pipe Peter Otten <__peter__@web.de> - 2016-03-10 23:09 +0100
    Re: Other difference with Perl: Python scripts in a pipe Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-10 15:16 -0700
      Re: Other difference with Perl: Python scripts in a pipe Fillmore <fillmore_remove@hotmail.com> - 2016-03-10 18:48 -0500
        Re: Other difference with Perl: Python scripts in a pipe INADA Naoki <songofacandy@gmail.com> - 2016-03-11 09:08 +0900
          Re: Other difference with Perl: Python scripts in a pipe Fillmore <fillmore_remove@hotmail.com> - 2016-03-10 19:26 -0500
            Re: Other difference with Perl: Python scripts in a pipe Alan Bawden <alan@csail.mit.edu> - 2016-03-11 01:45 -0500
              Re: Other difference with Perl: Python scripts in a pipe Jussi Piitulainen <jussi.piitulainen@helsinki.fi> - 2016-03-11 12:15 +0200
            Re: Other difference with Perl: Python scripts in a pipe Ethan Furman <ethan@stoneleaf.us> - 2016-03-11 00:55 -0800

#104553 — Other difference with Perl: Python scripts in a pipe

when I put a Python script in pipe with other commands, it will refuse 
to let go silently. Any way I can avoid this?

$ python somescript.py | head -5
line 1
line 3
line 3
line 4
line 5
Traceback (most recent call last):
   File "./somescript.py", line 50, in <module>
     sys.stdout.write(row[0])
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' 
encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

thanks

[toc] | [next] | [standalone]

#104554

On Thu, Mar 10, 2016 at 2:33 PM, Fillmore <fillmore_remove@hotmail.com> wrote:
>
> when I put a Python script in pipe with other commands, it will refuse to
> let go silently. Any way I can avoid this?

What is your script doing? I don't see this problem.

ikelly@queso:~ $ cat somescript.py
import sys

for i in range(20):
    sys.stdout.write('line %d\n' % i)
ikelly@queso:~ $ python somescript.py | head -5
line 0
line 1
line 2
line 3
line 4
ikelly@queso:~ $ python3 somescript.py | head -5
line 0
line 1
line 2
line 3
line 4

[toc] | [prev] | [next] | [standalone]

#104556

On 3/10/2016 4:46 PM, Ian Kelly wrote:
> On Thu, Mar 10, 2016 at 2:33 PM, Fillmore <fillmore_remove@hotmail.com> wrote:
>>
>> when I put a Python script in pipe with other commands, it will refuse to
>> let go silently. Any way I can avoid this?
>
> What is your script doing? I don't see this problem.
>
> ikelly@queso:~ $ cat somescript.py
> import sys
>
> for i in range(20):
>      sys.stdout.write('line %d\n' % i)

you are right. it's the with block :(

import sys
import csv

with open("somefile.tsv", newline='') as csvfile:

     myReader = csv.reader(csvfile, delimiter='\t')
     for row in myReader:

         for i in range(20):
             sys.stdout.write('line %d\n' % i)

$ ./somescript.py | head -5
line 0
line 1
line 2
line 3
line 4
Traceback (most recent call last):
   File "./somescript.py", line 12, in <module>
     sys.stdout.write('line %d\n' % i)
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' 
encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

[toc] | [prev] | [next] | [standalone]

#104557

Ian Kelly wrote:

> On Thu, Mar 10, 2016 at 2:33 PM, Fillmore <fillmore_remove@hotmail.com>
> wrote:
>>
>> when I put a Python script in pipe with other commands, it will refuse to
>> let go silently. Any way I can avoid this?
> 
> What is your script doing? I don't see this problem.
> 
> ikelly@queso:~ $ cat somescript.py
> import sys
> 
> for i in range(20):
>     sys.stdout.write('line %d\n' % i)
> ikelly@queso:~ $ python somescript.py | head -5
> line 0
> line 1
> line 2
> line 3
> line 4
> ikelly@queso:~ $ python3 somescript.py | head -5
> line 0
> line 1
> line 2
> line 3
> line 4

I suppose you need to fill the OS-level cache:

$ cat somescript.py 
import sys

for i in range(int(sys.argv[1])):
    sys.stdout.write('line %d\n' % i)
$ python somescript.py 20 | head -n5
line 0
line 1
line 2
line 3
line 4
$ python somescript.py 200 | head -n5
line 0
line 1
line 2
line 3
line 4
$ python somescript.py 2000 | head -n5
line 0
line 1
line 2
line 3
line 4
Traceback (most recent call last):
  File "somescript.py", line 4, in <module>
    sys.stdout.write('line %d\n' % i)
IOError: [Errno 32] Broken pipe

During my experiments I even got

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

occasionally.

[toc] | [prev] | [next] | [standalone]

#104558

On Thu, Mar 10, 2016 at 3:09 PM, Peter Otten <__peter__@web.de> wrote:
> I suppose you need to fill the OS-level cache:
>
> $ cat somescript.py
> import sys
>
> for i in range(int(sys.argv[1])):
>     sys.stdout.write('line %d\n' % i)
> $ python somescript.py 20 | head -n5
> line 0
> line 1
> line 2
> line 3
> line 4
> $ python somescript.py 200 | head -n5
> line 0
> line 1
> line 2
> line 3
> line 4
> $ python somescript.py 2000 | head -n5
> line 0
> line 1
> line 2
> line 3
> line 4
> Traceback (most recent call last):
>   File "somescript.py", line 4, in <module>
>     sys.stdout.write('line %d\n' % i)
> IOError: [Errno 32] Broken pipe
>
> During my experiments I even got
>
> close failed in file object destructor:
> sys.excepthook is missing
> lost sys.stderr
>
> occasionally.

Interesting, both of these are probably worth bringing up as issues on
the bugs.python.org tracker. I'm not sure that the behavior should be
changed (if we get an error, we shouldn't just swallow it) but it does
seem like a significant hassle for writing command-line
text-processing tools.

[toc] | [prev] | [next] | [standalone]

#104562

On 3/10/2016 5:16 PM, Ian Kelly wrote:
>
> Interesting, both of these are probably worth bringing up as issues on
> the bugs.python.org tracker. I'm not sure that the behavior should be
> changed (if we get an error, we shouldn't just swallow it) but it does
> seem like a significant hassle for writing command-line
> text-processing tools.

is it possible that I am the first one encountering this kind of issues?

[toc] | [prev] | [next] | [standalone]

#104565

On Fri, Mar 11, 2016 at 8:48 AM, Fillmore <fillmore_remove@hotmail.com>
wrote:

> On 3/10/2016 5:16 PM, Ian Kelly wrote:
>
>>
>> Interesting, both of these are probably worth bringing up as issues on
>> the bugs.python.org tracker. I'm not sure that the behavior should be
>> changed (if we get an error, we shouldn't just swallow it) but it does
>> seem like a significant hassle for writing command-line
>> text-processing tools.
>>
>
> is it possible that I am the first one encountering this kind of issues?
>
>
No.  I see it usually.

Python's zen says:

> Errors should never pass silently.
>    Unless explicitly silenced.

When failed to write to stdout, Python should raise Exception.
You can silence explicitly when it's safe:

try:
    print(...)
except BrokenPipeError:
    os.exit(0)

[toc] | [prev] | [next] | [standalone]

#104567

On 3/10/2016 7:08 PM, INADA Naoki wrote:

> No.  I see it usually.
>
> Python's zen says:
>
>> Errors should never pass silently.
>>     Unless explicitly silenced.
>
> When failed to write to stdout, Python should raise Exception.
> You can silence explicitly when it's safe:
>
> try:
>      print(...)
> except BrokenPipeError:
>      os.exit(0)
>

I don't like it. It makes Python not so good for command-line utilities

[toc] | [prev] | [next] | [standalone]

#104586

Fillmore <fillmore_remove@hotmail.com> writes:

> On 3/10/2016 7:08 PM, INADA Naoki wrote:
...
> I don't like it. It makes Python not so good for command-line utilities
>

You can easily restore the standard Unix command-line-friendly behavior
by doing:

    import signal
    signal.signal(signal.SIGINT, signal.SIG_DFL)
    signal.signal(signal.SIGPIPE, signal.SIG_DFL)

-- 
Alan Bawden

[toc] | [prev] | [next] | [standalone]

#104606

Alan Bawden writes:
> Fillmore writes:
>
>> On 3/10/2016 7:08 PM, INADA Naoki wrote:
> ...
>> I don't like it. It makes Python not so good for command-line utilities
>>
>
> You can easily restore the standard Unix command-line-friendly behavior
> by doing:
>
>     import signal
>     signal.signal(signal.SIGINT, signal.SIG_DFL)
>     signal.signal(signal.SIGPIPE, signal.SIG_DFL)

Thank you!

[toc] | [prev] | [next] | [standalone]

#104593

On 03/10/2016 04:26 PM, Fillmore wrote:
> On 3/10/2016 7:08 PM, INADA Naoki wrote:
>
>> No.  I see it usually.
>>
>> Python's zen says:
>>
>>> Errors should never pass silently.
>>>     Unless explicitly silenced.
>>
>> When failed to write to stdout, Python should raise Exception.
>> You can silence explicitly when it's safe:
>>
>> try:
>>      print(...)
>> except BrokenPipeError:
>>      os.exit(0)
>>
>
> I don't like it. It makes Python not so good for command-line utilities

You don't like typing two extra lines of code?

Or don't you like needing to understand what is going on so you know 
when to silence errors?

The try/except paradigm is very useful -- it's worth learning.

--
~Ethan~

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web