Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #107574 > unrolled thread

asyncio and subprocesses

Back to article view | Back to comp.lang.python

Contents

  asyncio and subprocesses David <dkeeney@travelbyroad.net> - 2016-04-24 15:07 -0700
    Re: asyncio and subprocesses Terry Reedy <tjreedy@udel.edu> - 2016-04-25 01:48 -0400

#107574 — asyncio and subprocesses

Is this a bug in the asyncio libraries?



This code:

'''
  proc = yield from asyncio.create_subprocess_exec(*cmd,
            stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)

  # read all data from subprocess pipe, copy to nextCoro
  ln = yield from proc.stdout.read(1024)
  while ln:
      yield from nextCoro.process(ln)
      ln = yield from proc.stdout.read(1024)

'''

will throw this exception:

Traceback (most recent call last):
File "/usr/project/bulk_aio.py", line 52, in db_source
ln = yield from proc.stdout.read(1024)
File "/usr/lib/python3.4/asyncio/streams.py", line 462, in read
self._maybe_resume_transport()
File "/usr/lib/python3.4/asyncio/streams.py", line 349, in _maybe_resume_transport  self._transport.resume_reading()
File "/usr/lib/python3.4/asyncio/unix_events.py", line 364, in resume_reading  self._loop.add_reader(self._fileno, self._read_ready)
AttributeError: 'NoneType' object has no attribute 'add_reader'


The exception always happens at the end of the subprocess's run, in what would be the last read. Whether it happens correlates with the time needed for nextCoro.process.  If each iteration takes more than about a millisecond, the exception will be thrown.

It *seems* that when the transport loses the pipe connection, it schedules the event loop for removal immediately, and the _loop gets set to None before the data can all be read.  This is the sequence in unix_events.py _UnixReadPipeTransport._read_ready.

Is there a workaround to avoid this exception?  Is this a fixed bug, already?  I am using Python 3.4.2 as distributed in Ubuntu Lucid, with built-in asyncio.

Thank you.
David

[toc] | [next] | [standalone]

#107581

On 4/24/2016 6:07 PM, David wrote:
>
> Is this a bug in the asyncio libraries?
> Is this a fixed bug, already?  I am using Python 3.4.2 as distributed in Ubuntu Lucid, with built-in asyncio.

The people who patch asyncio do not read this list.  Either install a 
current release or try the tulip release on Pypy (its purpose is to make 
asyncio available on versions before 3.4 but should also run on 3.4.  I 
don't know if it conflicts with the included asyncio.


-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web