Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #30869 > unrolled thread

try/except KeyError vs "if name in ..."

Back to article view | Back to comp.lang.python

Contents

  try/except KeyError vs "if name in ..." Manuel Pégourié-Gonnard <mpg@elzevir.fr> - 2012-10-06 08:27 +0200
    Re: try/except KeyError vs "if name in ..." Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-10-06 08:33 +0000
      Re: try/except KeyError vs "if name in ..." Manuel Pégourié-Gonnard <mpg@elzevir.fr> - 2012-10-06 12:08 +0200
    Re: try/except KeyError vs "if name in ..." "Günther Dietrich" <gd.usenet@spamfence.net> - 2012-10-06 10:49 +0200
      Re: try/except KeyError vs "if name in ..." Manuel Pégourié-Gonnard <mpg@elzevir.fr> - 2012-10-06 12:09 +0200
    Re: try/except KeyError vs "if name in ..." Dave Angel <d@davea.name> - 2012-10-06 07:36 -0400
    Re: try/except KeyError vs "if name in ..." Terry Reedy <tjreedy@udel.edu> - 2012-10-06 15:42 -0400
      Re: try/except KeyError vs "if name in ..." Ramchandra Apte <maniandram01@gmail.com> - 2012-10-06 21:17 -0700
      Re: try/except KeyError vs "if name in ..." Ramchandra Apte <maniandram01@gmail.com> - 2012-10-06 21:17 -0700

#30869 — try/except KeyError vs "if name in ..."

Hi,

I was looking at the example found here [1] which begins with:

[1] http://docs.python.org/py3k/library/imp.html#examples

def __import__(name, globals=None, locals=None, fromlist=None):
    # Fast path: see if the module has already been imported.
    try:
        return sys.modules[name]
    except KeyError:
        pass

I was wondering if the formulation

    if name in sys.modules:
        return sys.modules[name]

would be equivalent. IOW, is using try/except here only a matter of
style or a necessity?

I'm suspecting that maybe, in multithreaded environments, the second
option may be subject to a race condition, if another thread removes
name frome sys.modules between the if and the return, but as I'm not
very familiar (yet) with Python threads, I'm not sure it is a real
concern here.

And maybe there are other reasons I'm completely missing for prefering
EAFP over LBYL here?

Thanks in advance for your comments.

-- 
Manuel Pégourié-Gonnard - http://people.math.jussieu.fr/~mpg/

[toc] | [next] | [standalone]

#30874

On Sat, 06 Oct 2012 08:27:25 +0200, Manuel Pégourié-Gonnard wrote:

> Hi,
> 
> I was looking at the example found here [1] which begins with:
> 
> [1] http://docs.python.org/py3k/library/imp.html#examples
> 
> def __import__(name, globals=None, locals=None, fromlist=None):
>     # Fast path: see if the module has already been imported. try:
>         return sys.modules[name]
>     except KeyError:
>         pass
> 
> I was wondering if the formulation
> 
>     if name in sys.modules:
>         return sys.modules[name]
> 
> would be equivalent. IOW, is using try/except here only a matter of
> style or a necessity?

Mostly style, but not entirely.

If you expect that most of the time the module will be found, the 
try...except version will be faster. If you expect that most of the time 
the module will not be found, the "if name in" version will be faster.

But see also:

> I'm suspecting that maybe, in multithreaded environments, the second
> option may be subject to a race condition, if another thread removes
> name frome sys.modules between the if and the return, but as I'm not
> very familiar (yet) with Python threads, I'm not sure it is a real
> concern here.

In practice, no, it would be very unusual for another thread to remove 
the name from sys.modules. So don't do that :)

But in principle, yes, it is a race condition and yes it is a (small) 
concern. Since it is so easy to avoid even this tiny risk, why not use 
the try...except version and avoid it completely?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#30880

Steven D'Aprano scripsit :

> If you expect that most of the time the module will be found, the 
> try...except version will be faster. If you expect that most of the time 
> the module will not be found, the "if name in" version will be faster.
>
Ok.

In the particular case of __import__, I guess speed is not crucial since
I doubt import often happen within a program's inner loop. But I'll
remember that point for other cases anyway.

>> I'm suspecting that maybe, in multithreaded environments, the second
>> option may be subject to a race condition, if another thread removes
>> name frome sys.modules between the if and the return, but as I'm not
>> very familiar (yet) with Python threads, I'm not sure it is a real
>> concern here.
>
> In practice, no, it would be very unusual for another thread to remove 
> the name from sys.modules. So don't do that :)
>
That wasn't my intention. But sometimes other people may be "creative" :)

> But in principle, yes, it is a race condition and yes it is a (small) 
> concern. Since it is so easy to avoid even this tiny risk, why not use 
> the try...except version and avoid it completely?
>
Ok.

Thanks for your explanations.

-- 
Manuel Pégourié-Gonnard - http://people.math.jussieu.fr/~mpg/

[toc] | [prev] | [next] | [standalone]

#30875

Manuel Pégourié-Gonnard <mpg@elzevir.fr> wrote:

>Hi,
>I was looking at the example found here [1] which begins with:
>
>[1] http://docs.python.org/py3k/library/imp.html#examples
>
>def __import__(name, globals=None, locals=None, fromlist=None):
>    # Fast path: see if the module has already been imported.
>    try:
>        return sys.modules[name]
>    except KeyError:
>        pass
>
>I was wondering if the formulation
>
>    if name in sys.modules:
>        return sys.modules[name]
>
>would be equivalent. IOW, is using try/except here only a matter of
>style or a necessity?

Somewhere I read a text regarding 'try:' versus 'if'. If you take the 
probabitility into consideration, how many times the test will fail or 
succeed, there are two possibilities:
- If the test will fail only on very rare occasions: Use 'try:'. When 
the statement(s) in the try-path succeed, the try:/except: construct 
will not consume additional execution time (not even for the test). The 
statements will just be executed as they are. Only in the (rare) case of 
failure, the exception handling will take additional execution time (a 
considerably big amount). The fact, that in python it is not named 
'error handling', but 'exception handling' reflects this. The failure 
should be the exception, also in matters of occurrence.
- If the relation between success and failure is not predictable, or if 
the case of failure will be frequent, use 'if'. The failure of a 'try:' 
gives you a penalty in form of consumption of a high amount of execution 
time. So, in these constellations, it is better to accept the relatively 
small amount of execution time taken by the explicit test.

Obviously, you can use 'try:' only, if there is the possibility to 
produce an exception on failure. In the other cases you must use the 
explicit test by 'if'.


>I'm suspecting that maybe, in multithreaded environments, the second
>option may be subject to a race condition, if another thread removes
>name frome sys.modules between the if and the return, but as I'm not
>very familiar (yet) with Python threads, I'm not sure it is a real
>concern here.

Your idea sounds reasonable, but I also am not yet familiar with threads.


>And maybe there are other reasons I'm completely missing for prefering
>EAFP over LBYL here?

One reason might be what I described above.



Best regards,

Günther

[toc] | [prev] | [next] | [standalone]

#30882

Günther Dietrich scripsit :

> Somewhere I read a text regarding 'try:' versus 'if'. If you take the 
> probabitility into consideration, how many times the test will fail or 
> succeed, there are two possibilities: [...]

Ok, thanks for the details!

-- 
Manuel Pégourié-Gonnard - http://people.math.jussieu.fr/~mpg/

[toc] | [prev] | [next] | [standalone]

#30885

On 10/06/2012 02:27 AM, Manuel Pégourié-Gonnard wrote:
> Hi,
>
> I was looking at the example found here [1] which begins with:
>
> [1] http://docs.python.org/py3k/library/imp.html#examples
>
> def __import__(name, globals=None, locals=None, fromlist=None):
>     # Fast path: see if the module has already been imported.
>     try:
>         return sys.modules[name]
>     except KeyError:
>         pass
>
> I was wondering if the formulation
>
>     if name in sys.modules:
>         return sys.modules[name]
>
> would be equivalent. IOW, is using try/except here only a matter of
> style or a necessity?
>
> I'm suspecting that maybe, in multithreaded environments, the second
> option may be subject to a race condition, if another thread removes
> name frome sys.modules between the if and the return, but as I'm not
> very familiar (yet) with Python threads, I'm not sure it is a real
> concern here.
>
> And maybe there are other reasons I'm completely missing for prefering
> EAFP over LBYL here?
>
> Thanks in advance for your comments.
>

Guidelines for writing library code may very well be different than for
writing your application.  And if your application is trying to do
something similar with *import*, chances are that it's calling a library
function that already starts with the test against sys.modules.  So if
this is an application question, the answer is probably "don't do either
one, just do the import, checking for the exceptions that it may throw."

The distinction in performance between the success and failure modes of
the try/catch isn't nearly as large as one of the other responses might
lead you to believe.  For example, a for loop generally terminates with
a raise (of StopIteration exception), and that doesn't convince us to
replace it with a while loop.  Besides, in this case, the except code
effectively includes the entire import, which would completely swamp the
overhead of the raise.

If we assume the question was more generally about EAFT vs. LBYL, and
not just about the case of accessing the system data structure
sys.modules, then the issues change somewhat.

If we do a LBYL, we have to know that we've covered all interesting
cases with our test.  Multithreading is one case where we can get a race
condition.  There are times when we might be able to know either that
there are not other threads, or that the other threads don't mess with
the stuff we're testing.  For example, there are enough problems with
import and threads that we might just have a development policy that (in
this program) we will do all our imports before starting any additional
threads, and that we will never try to unload an import, single threaded
or not.  But for other conditions, we might be affected either by the
system or by other processes within it.  Or even affected by other
asynchronous events over a network.

If we do an EAFP, then we have to figure out what exceptions are
possible.  However, adding more exceptions with different treatments is
quite easy, and they don't all have to be done at the same level.  Some
may be left for our caller to deal with.  I think the major thing that
people mind about try/catch is that it seems to break up the program
flow.  However, that paradigm grows on you as you get accustomed to it.

-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#30898

On 10/6/2012 7:36 AM, Dave Angel wrote:

> The distinction in performance between the success and failure modes of
> the try/catch isn't nearly as large as one of the other responses might
> lead you to believe.  For example, a for loop generally terminates with
> a raise (of StopIteration exception), and that doesn't convince us to
> replace it with a while loop.

For statement generally loop many times, up to millions of times, 
without an exception being raised, whereas while statements test the 
condition each time around the loop. So the rule 'if failure is rare 
(less than 10-20%) use try', applies here. For if/them versus 
try/except, I don't worry too much about it.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#30907

On Sunday, 7 October 2012 01:12:56 UTC+5:30, Terry Reedy  wrote:
> On 10/6/2012 7:36 AM, Dave Angel wrote:
> 
> 
> 
> > The distinction in performance between the success and failure modes of
> 
> > the try/catch isn't nearly as large as one of the other responses might
> 
> > lead you to believe.  For example, a for loop generally terminates with
> 
> > a raise (of StopIteration exception), and that doesn't convince us to
> 
> > replace it with a while loop.
> 
> 
> 
> For statement generally loop many times, up to millions of times, 
> 
> without an exception being raised, whereas while statements test the 
> 
> condition each time around the loop. So the rule 'if failure is rare 
> 
> (less than 10-20%) use try', applies here. For if/them versus 
> 
> try/except, I don't worry too much about it.
> 
> 
> 
> -- 
> 
> Terry Jan Reedy

I use try and except when I need to raise exceptions e.g.:

try:
    value = vm.variables[name]
except KeyError:
    raise NameError("variable name not defined in VM's variables")

[toc] | [prev] | [next] | [standalone]

#30908

On Sunday, 7 October 2012 01:12:56 UTC+5:30, Terry Reedy  wrote:
> On 10/6/2012 7:36 AM, Dave Angel wrote:
> 
> 
> 
> > The distinction in performance between the success and failure modes of
> 
> > the try/catch isn't nearly as large as one of the other responses might
> 
> > lead you to believe.  For example, a for loop generally terminates with
> 
> > a raise (of StopIteration exception), and that doesn't convince us to
> 
> > replace it with a while loop.
> 
> 
> 
> For statement generally loop many times, up to millions of times, 
> 
> without an exception being raised, whereas while statements test the 
> 
> condition each time around the loop. So the rule 'if failure is rare 
> 
> (less than 10-20%) use try', applies here. For if/them versus 
> 
> try/except, I don't worry too much about it.
> 
> 
> 
> -- 
> 
> Terry Jan Reedy

I use try and except when I need to raise exceptions e.g.:

try:
    value = vm.variables[name]
except KeyError:
    raise NameError("variable name not defined in VM's variables")

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web