Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #85269 > unrolled thread

multiprocessing.Queue() and missing sem_open

Back to article view | Back to comp.lang.python

Contents

  multiprocessing.Queue() and missing sem_open ole-usenet-spam@gmx.net (Оlе Ѕtrеісhеr) - 2015-02-05 21:22 +0100
    Re: multiprocessing.Queue() and missing sem_open Chris Angelico <rosuav@gmail.com> - 2015-02-06 09:29 +1100
      Re: multiprocessing.Queue() and missing sem_open ole-usenet-spam@gmx.net (Оlе Ѕtrеісhеr) - 2015-02-06 09:27 +0100
        Re: multiprocessing.Queue() and missing sem_open Chris Angelico <rosuav@gmail.com> - 2015-02-06 19:34 +1100

#85269 — multiprocessing.Queue() and missing sem_open

Hi,

I am just trying to prepare a package (astropy) for (Debian) Hurd. This
os lacks a sem_open() implementation. When I now try:

import multiprocessing
q = multiprocessing.Queue()

I get an ImportError with Python 2.7, but an AttributeError with Python
3.4. In the documentation of multiprocessing.Queue() I couldn't find any
hint that it would throw this exception.

I am now curious, that

1. this behaviour is not documented
2. it changed at some point, without documentation.

Why does it not return a NotImplementedError (this is what I would
expect if a function is not implemented by the OS)?

Can I be sure that the following works also in future?

try
    q = multiprocessing.Queue()
except (ImportError, AttributeError)
    # handle the case of missing sem_open

Or what is the correct way to catch a not working Queue caused by a
missing sem_open() implementation?

Best regards

Ole

[toc] | [next] | [standalone]

#85273

On Fri, Feb 6, 2015 at 7:22 AM, Оlе Ѕtrеісhеr <ole-usenet-spam@gmx.net> wrote:
> I am just trying to prepare a package (astropy) for (Debian) Hurd. This
> os lacks a sem_open() implementation. When I now try:
>
> import multiprocessing
> q = multiprocessing.Queue()
>
> I get an ImportError with Python 2.7, but an AttributeError with Python
> 3.4. In the documentation of multiprocessing.Queue() I couldn't find any
> hint that it would throw this exception.

Neither of those errors is being consciously thrown by the Queue
class; the latter means that multiprocessing exists but it has no
Queue in it, and the former means that the entire multiprocessing
module is absent.

I'm guessing you built Python from source? If so, you should have had
a report at the bottom of the build output stating which of the
optional packages weren't built, which on 2.7 will have included
multiprocessing. You might be able to install some dependency and get
that to build, though I don't know what it would be, given that the
3.4 build managed to find it.

But this is the exact way that Python will normally signal that
something isn't available. If your platform doesn't have, say,
os.O_BINARY, because that flag has no meaning for you, then you don't
get NotImplementedError - you simply get AttributeError.

> Can I be sure that the following works also in future?
>
> try
>     q = multiprocessing.Queue()
> except (ImportError, AttributeError)
>     # handle the case of missing sem_open
>
> Or what is the correct way to catch a not working Queue caused by a
> missing sem_open() implementation?

The ImportError will come from the import statement, the
AttributeError will come from the attribute lookup - the above code
can't[1] bomb out with ImportError. So a more correct way would be
something like:

try:
    import multiprocessing
    multiprocessing.Queue
except (ImportError, AttributeError):
    # handle the absence

Or alternatively, don't even bother to try/except the import. If your
code requires multiprocessing.Queue (as opposed to just queue.Queue),
chances are you can't cope with the absence of the module.

ChrisA

[1] Yeah yeah, anything can do anything. You know what I mean.

[toc] | [prev] | [next] | [standalone]

#85282

Chris Angelico <rosuav@gmail.com> writes:

> On Fri, Feb 6, 2015 at 7:22 AM, Оlе Ѕtrеісhеr <ole-usenet-spam@gmx.net> wrote:
>> I am just trying to prepare a package (astropy) for (Debian) Hurd. This
>> os lacks a sem_open() implementation. When I now try:
>>
>> import multiprocessing
>> q = multiprocessing.Queue()
>>
>> I get an ImportError with Python 2.7, but an AttributeError with Python
>> 3.4. In the documentation of multiprocessing.Queue() I couldn't find any
>> hint that it would throw this exception.
>
> Neither of those errors is being consciously thrown by the Queue
> class; the latter means that multiprocessing exists but it has no
> Queue in it, and the former means that the entire multiprocessing
> module is absent.

Trace with 3.4.2:

self = <multiprocessing.context.DefaultContext object at 0x1449b0c>, maxsize = 0

    def Queue(self, maxsize=0):
        '''Returns a queue object'''
        from .queues import Queue
>       return Queue(maxsize, ctx=self.get_context())

/usr/lib/python3.4/multiprocessing/context.py:101: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.queues.Queue object at 0x511e8cc>, maxsize = 0

    def __init__(self, maxsize=0, *, ctx):
        if maxsize <= 0:
>           maxsize = _multiprocessing.SemLock.SEM_VALUE_MAX
E           AttributeError: 'module' object has no attribute 'SemLock'

/usr/lib/python3.4/multiprocessing/queues.py:38: AttributeError

Trace with 2.7.9:

maxsize = 0

    def Queue(maxsize=0):
        '''
        Returns a queue object
        '''
>       from multiprocessing.queues import Queue

/usr/lib/python2.7/multiprocessing/__init__.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    __all__ = ['Queue', 'SimpleQueue', 'JoinableQueue']
    
    import sys
    import os
    import threading
    import collections
    import time
    import atexit
    import weakref
    
    from Queue import Empty, Full
    import _multiprocessing
    from multiprocessing import Pipe
>   from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition

/usr/lib/python2.7/multiprocessing/queues.py:48: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Condition', 'Event'
    ]
    
    import threading
    import os
    import sys
    
    from time import time as _time, sleep as _sleep
    
    import _multiprocessing
    from multiprocessing.process import current_process
    from multiprocessing.util import Finalize, register_after_fork, debug
    from multiprocessing.forking import assert_spawning, Popen
    
    # Try to import the mp.synchronize module cleanly, if it fails
    # raise ImportError for platforms lacking a working sem_open implementation.
    # See issue 3770
    try:
    from _multiprocessing import SemLock
    except (ImportError):
    raise ImportError("This platform lacks a functioning sem_open" +
                      " implementation, therefore, the required" +
                      " synchronization primitives needed will not" +
>                     " function, see issue 3770.")
E                     ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

/usr/lib/python2.7/multiprocessing/synchronize.py:59: ImportError

Both are with the same Hurd version.

> I'm guessing you built Python from source?

No, I am using the precompiled version that comes with Debian. It has no
specific patch for sem_open.

> But this is the exact way that Python will normally signal that
> something isn't available. If your platform doesn't have, say,
> os.O_BINARY, because that flag has no meaning for you, then you don't
> get NotImplementedError - you simply get AttributeError.

As you can see from the trace, multiprocessing.Queue is there, but
initialization fails.

>> Can I be sure that the following works also in future?
>>
>> try
>>     q = multiprocessing.Queue()
>> except (ImportError, AttributeError)
>>     # handle the case of missing sem_open
>>
>> Or what is the correct way to catch a not working Queue caused by a
>> missing sem_open() implementation?
>
> The ImportError will come from the import statement

No.

> the AttributeError will come from the attribute lookup - the above code
> can't[1] bomb out with ImportError.

Maybe it can't but it actually does.

> So a more correct way would be something like:
>
> try:
>     import multiprocessing
>     multiprocessing.Queue
> except (ImportError, AttributeError):
>     # handle the absence

Is it sure that this will work in the future as well?

> Or alternatively, don't even bother to try/except the import. If your
> code requires multiprocessing.Queue (as opposed to just queue.Queue),
> chances are you can't cope with the absence of the module.

This code is used for a fast, asynchronious reader. If it is not
implemented, there is a (slower) fallback solution. However, we just want
to catch the "is not implemented" case here but not any other problem
that may appear during opening or reading the file.

Shall I file a bug for this?

Best

Ole

[toc] | [prev] | [next] | [standalone]

#85283

On Fri, Feb 6, 2015 at 7:27 PM, Оlе Ѕtrеісhеr <ole-usenet-spam@gmx.net> wrote:
>> the AttributeError will come from the attribute lookup - the above code
>> can't[1] bomb out with ImportError.
>
> Maybe it can't but it actually does.

Huh. Okay, my bad. (I don't have a Hurd system to test on, all my
Debians are Linux.) Sorry for the misinformation.

>> So a more correct way would be something like:
>>
>> try:
>>     import multiprocessing
>>     multiprocessing.Queue
>> except (ImportError, AttributeError):
>>     # handle the absence
>
> Is it sure that this will work in the future as well?

Frankly, I don't know now. It's reasonably likely that it will, but in
the absence of actual documentation, I couldn't say.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web