Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #57806 > unrolled thread

how to avoid checking the same condition repeatedly ?

Back to article view | Back to comp.lang.python

Contents

  how to avoid checking the same condition repeatedly ? Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> - 2013-10-28 09:50 +0000
    Re: how to avoid checking the same condition repeatedly ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-10-28 11:14 +0000
      Re: how to avoid checking the same condition repeatedly ? Chris Angelico <rosuav@gmail.com> - 2013-10-28 22:18 +1100
        Re: how to avoid checking the same condition repeatedly ? Steven D'Aprano <steve@pearwood.info> - 2013-10-29 04:51 +0000
    Re: how to avoid checking the same condition repeatedly ? Piet van Oostrum <piet@vanoostrum.org> - 2013-10-28 14:31 -0400
    Re: how to avoid checking the same condition repeatedly ? Nobody <nobody@nowhere.com> - 2013-10-28 19:37 +0000
      Re: how to avoid checking the same condition repeatedly ? Neil Cerutti <neilc@norwich.edu> - 2013-10-29 13:44 +0000
        Re: how to avoid checking the same condition repeatedly ? rusi <rustompmody@gmail.com> - 2013-10-29 09:13 -0700
    Re: how to avoid checking the same condition repeatedly ? Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-28 23:14 -0700
      Re: how to avoid checking the same condition repeatedly ? Chris Angelico <rosuav@gmail.com> - 2013-10-29 17:39 +1100
    Re: how to avoid checking the same condition repeatedly ? Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-28 23:53 -0700
      Re: how to avoid checking the same condition repeatedly ? Chris Angelico <rosuav@gmail.com> - 2013-10-29 18:09 +1100
    Re: how to avoid checking the same condition repeatedly ? Peter Cacioppi <peter.cacioppi@gmail.com> - 2013-10-29 16:55 -0700
    Re: how to avoid checking the same condition repeatedly ? alex23 <wuwei23@gmail.com> - 2013-10-30 11:38 +1000
    Re: how to avoid checking the same condition repeatedly ? Mariano Anaya <marianoanaya@gmail.com> - 2013-10-30 06:34 -0700

#57806 — how to avoid checking the same condition repeatedly ?

Dear all,
this is a recurring programming problem that I'm just not sure how to solve
optimally, so I thought I'd ask for your advice:
imagine you have a flag set somewhere earlier in your code, e.g.,

needs_processing = True

then in a for loop you're processing the elements of an iterable, but the
kind of processing depends on the flag, e.g.,:

for elem in iterable:
    if needs_processing:
        pre_process(elem)  # reformat elem in place
    print(elem)

this checks the condition every time through the for loop, even though there
is no chance for needs_processing to change inside the loop, which does not
look very efficient. Of course, you could rewrite the above as:

if needs_processing:
    for elem in iterable:
        pre_process(elem)  # reformat elem in place
        print(elem)
else:
    for elem in iterable:
        print(elem)

but this means unnecessary code-duplication.

You could also define functions (or class methods):
def pre_process_and_print (item):
    pre_process(item)
    print(item)

def raw_print (item):
    print(item)

then:
process = pre_process_and_print if needs_processing else raw_print
for elem in iterable:
    process(elem)

but while this works for the simple example here, it becomes complicated if
pre_process requires more information to do its job because then you will
have to start passing around (potentially lots of) arguments.

So my question is: is there an agreed-upon generally best way of dealing
with this?

Thanks for your help,
Wolfgang

[toc] | [next] | [standalone]

#57810

On Mon, 28 Oct 2013 09:50:19 +0000, Wolfgang Maier wrote:

> Dear all,
> this is a recurring programming problem that I'm just not sure how to
> solve optimally, so I thought I'd ask for your advice: imagine you have
> a flag set somewhere earlier in your code, e.g.,
> 
> needs_processing = True
> 
> then in a for loop you're processing the elements of an iterable, but
> the kind of processing depends on the flag, e.g.,:
> 
> for elem in iterable:
>     if needs_processing:
>         pre_process(elem)  # reformat elem in place
>     print(elem)
> 
> this checks the condition every time through the for loop, even though
> there is no chance for needs_processing to change inside the loop, which
> does not look very efficient. 

If there is absolutely no chance the flag could change, then the best way 
is to test the flag once:

> if needs_processing:
>     for elem in iterable:
>         pre_process(elem)  # reformat elem in place 
>         print(elem)
> else:
>     for elem in iterable:
>         print(elem)


The amount of duplication here is trivial. Sometimes you can avoid even 
that:

if needs_preprocessing:
    preprocess(sequence)
process(sequence)


This is especially useful if you can set up a pipeline of generators:


if flag1:
    data = (preprocess(item) for item in data)
if flag2:
    data = (process(item) for item in data)
if flag3:
    data = (postprocess(item) for item in data)
for item in data:
    print(item)



Another option is to use a factory-function:

def factory(flag):
    def process(item):
        do_this(item)
        do_that(item)
        do_something_else(item)
        print(item)
    if flag:
        def func(item):
            preprocess(item)
            process(item)
        return func
    else:
        return process



func = factory(needs_preprocessing)
for elem in iterable:
    func(elem)




> So my question is: is there an agreed-upon generally best way of dealing
> with this?

No. It's mostly a matter of taste. Those coming from a Java background 
will probably prefer the factory or builder idiom. Those coming from a 
functional language background, like Haskell or Lisp, or with experience 
with Unix pipes, will probably prefer using chained generators.

The only solution you absolutely should avoid is the naive "copy and 
paste" solution, where the amount of code duplication is extensive:


if needs_preprocessing:
    for elem in iterable:
        # If you change this block, you must remember to change the 
        # block below too!
        preprocess(item)
        do_this(item)
        do_that(item)
        do_something_else(item, arg1, arg2, expression)
        do_another_thing(item + arg3, key=whatever)
        print(item)
else:
    for elem in iterable:
        # If you change this block, you must remember to change the 
        # block above too!
        preprocess(item)
        do_this(item)
        do_that(item)
        do_something_else(item, arg1, arg2, expression)
        do_another_thing(item, key=whatever)
        print(item)


The careful reader will notice that there's already a bug in this.


Even checking the variable inside the loop is okay, provided you don't 
care too much about performance, although I wouldn't call it "best 
practice". Merely "acceptable for code that isn't performance-critical".


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#57811

On Mon, Oct 28, 2013 at 10:14 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> if needs_preprocessing:
>     for elem in iterable:
>         # If you change this block, you must remember to change the
>         # block below too!
>         preprocess(item)
>         do_this(item)
>         do_that(item)
>         do_something_else(item, arg1, arg2, expression)
>         do_another_thing(item + arg3, key=whatever)
>         print(item)
> else:
>     for elem in iterable:
>         # If you change this block, you must remember to change the
>         # block above too!
>         preprocess(item)
>         do_this(item)
>         do_that(item)
>         do_something_else(item, arg1, arg2, expression)
>         do_another_thing(item, key=whatever)
>         print(item)
>
>
> The careful reader will notice that there's already a bug in this.

For a start, you're preprocessing in the second block... I don't think
that was your intentional bug :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#57878

On Mon, 28 Oct 2013 22:18:45 +1100, Chris Angelico wrote:

> On Mon, Oct 28, 2013 at 10:14 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> if needs_preprocessing:
>>     for elem in iterable:
>>         # If you change this block, you must remember to change the #
>>         block below too!
>>         preprocess(item)
>>         do_this(item)
>>         do_that(item)
>>         do_something_else(item, arg1, arg2, expression)
>>         do_another_thing(item + arg3, key=whatever) print(item)
>> else:
>>     for elem in iterable:
>>         # If you change this block, you must remember to change the #
>>         block above too!
>>         preprocess(item)
>>         do_this(item)
>>         do_that(item)
>>         do_something_else(item, arg1, arg2, expression)
>>         do_another_thing(item, key=whatever)
>>         print(item)
>>
>>
>> The careful reader will notice that there's already a bug in this.
> 
> For a start, you're preprocessing in the second block... I don't think
> that was your intentional bug :)

Heh, no, that wasn't the intentional bug. But it does go to show the 
risks of copy-and-paste programming.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#57846

Wolfgang Maier <wolfgang.maier@biologie.uni-freiburg.de> writes:

> Dear all,
> this is a recurring programming problem that I'm just not sure how to solve
> optimally, so I thought I'd ask for your advice:
> imagine you have a flag set somewhere earlier in your code, e.g.,
>
> needs_processing = True
>
> then in a for loop you're processing the elements of an iterable, but the
> kind of processing depends on the flag, e.g.,:
>
> for elem in iterable:
>     if needs_processing:
>         pre_process(elem)  # reformat elem in place
>     print(elem)
>
> this checks the condition every time through the for loop, even though there
> is no chance for needs_processing to change inside the loop, which does not
> look very efficient. 

I bet in most cases you won't notice the time used to check the condition.
Beware of premature optimization!
-- 
Piet van Oostrum <piet@vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]

[toc] | [prev] | [next] | [standalone]

#57848

On Mon, 28 Oct 2013 09:50:19 +0000, Wolfgang Maier wrote:

> So my question is: is there an agreed-upon generally best way of dealing
> with this?

Yes. Just leave the test inside the loop.

If you're sufficiently concerned about performance that you're willing to
trade clarity for it, you shouldn't be using Python in the first place.

[toc] | [prev] | [next] | [standalone]

#57915

On 2013-10-28, Nobody <nobody@nowhere.com> wrote:
> On Mon, 28 Oct 2013 09:50:19 +0000, Wolfgang Maier wrote:
>> So my question is: is there an agreed-upon generally best way
>> of dealing with this?
>
> Yes. Just leave the test inside the loop.
>
> If you're sufficiently concerned about performance that you're
> willing to trade clarity for it, you shouldn't be using Python
> in the first place.

When you detect a code small, as Wolfgang did, e.g., "I'm
repeating the same exact test condition in several places," you
should not simply ignore it, even in Python.

-- 
Neil Cerutti

[toc] | [prev] | [next] | [standalone]

#57926

On Tuesday, October 29, 2013 7:14:51 PM UTC+5:30, Neil Cerutti wrote:
> On 2013-10-28, Nobody  wrote:
> > If you're sufficiently concerned about performance that you're
> > willing to trade clarity for it, you shouldn't be using Python
> > in the first place.
> 
> 
> When you detect a code small, as Wolfgang did, e.g., "I'm
> repeating the same exact test condition in several places," you
> should not simply ignore it, even in Python.

Yes
It is an agenda of functional programming that such programs should be modularly writeable without loss of performance:
http://homepages.inf.ed.ac.uk/wadler/topics/deforestation.html
[the trees there include lists]

Unfortunately for the imperative programmer, some forms of modularization are 
simply not conceivable that are natural for functional programmers (or as 
Steven noted shell-script writers). eg The loop:

while pred:
  statement
is simply one unit and cannot be split further.

Whereas in a FPL one can write:
iterate statement  >>>  takeWhile pred
-- the >>> being analogous to | in bash

[toc] | [prev] | [next] | [standalone]

#57886

Nobody (yes, his name is Nobody) said:

"If you're sufficiently concerned about performance that you're willing to
trade clarity for it, you shouldn't be using Python in the first place."

I think the correct thing to say here is, IF you know this subroutine is a bottleneck, THEN probably this subroutine (or even the module it lives within) should be recoding in a language closer to the metal (like C).

I don't think it's correct to imply that people very concerned about performance should not use Python. (And I agree, Nobody implied that ;) But sometimes performance concerns require the bottleneck(s) be recoded in a manner that sacrifices readability for performance, to include a different language. Python generally plays well with other languages, no? So code it in Py, profile it, refactor the bottlenecks as needed.

[toc] | [prev] | [next] | [standalone]

#57888

On Tue, Oct 29, 2013 at 5:14 PM, Peter Cacioppi
<peter.cacioppi@gmail.com> wrote:
> Nobody (yes, his name is Nobody) said:
>
> "If you're sufficiently concerned about performance that you're willing to
> trade clarity for it, you shouldn't be using Python in the first place."
>
> I don't think it's correct to imply that people very concerned about performance should not use Python. (And I agree, Nobody implied that ;)

No, I don't think he implied that. You can care about performance
while still putting code clarity as a higher priority :) If you
actually profile and find that something-or-other is a bottleneck,
chances are you can break it out into a function with minimal loss of
clarity, and then reimplement that function in C (maybe wielding
Cython for the main work). That doesn't compromise clarity.
Duplicating a loop to hoist a condition _does_. Of course, in C, you
can let the compiler do it for you, but Python can't be sure that it's
as constant as you think, so it can't be changed.

ChrisA

[toc] | [prev] | [next] | [standalone]

#57889

Chris said

" If you actually profile and find that something-or-other is a bottleneck,
chances are you can break it out into a function with minimal loss of
clarity, and then reimplement that function in C (maybe wielding
Cython for the main work). That doesn't compromise clarity. "

Well, I'm not going to go back and forth saying "does too, does not" with you. I have a 7 year old for those sorts of arguments. 

And I think we are saying more or less the same thing.

I agree that isolating your bottleneck to the tightest possible subroutine usually doesn't compromise clarity. 

But once you are re-implementing the bottleneck in a different language.... esp a language  notorious for high performance nuggets of opaqueness... that does compromise clarity, to some extent.

(Does not, does too, does not, does too, ok we're done with that part)

But this sort of bottleneck refactoring can be done in a careful way that minimizes the damage to readability. And a strength of py is it tends to encourage this "as pretty as possible" approach to bottleneck refactoring.

This is what you're saying, right?

[toc] | [prev] | [next] | [standalone]

#57891

On Tue, Oct 29, 2013 at 5:53 PM, Peter Cacioppi
<peter.cacioppi@gmail.com> wrote:
> But this sort of bottleneck refactoring can be done in a careful way that minimizes the damage to readability. And a strength of py is it tends to encourage this "as pretty as possible" approach to bottleneck refactoring.
>
> This is what you're saying, right?

Yep, that's about the size of it. Want some examples of what costs no
clarity to reimplement in another language? Check out the Python
standard library. Some of that is implemented in C (in CPython) and
some in Python, and you can't tell and needn't care which. Code
clarity isn't hurt, because those functions would be named functions
even without.

ChrisA

[toc] | [prev] | [next] | [standalone]

#57983

Chris said :
"Want some examples of what costs no clarity to reimplement in another language? Check out the Python standard library. Some of that is implemented in C (in CPython) and some in Python, and you can't tell and needn't care which."

To ME (a consumer of the CPython library) there is zero cost to clarity.

To the angels that maintain/develop this library and need to go inside the black box regularly .... there is a non-zero cost to clarity.

Right? 

(I'd rather run the risk of stating the obvious than missing something clever, that's why I keep hitting this sessile equine).

[toc] | [prev] | [next] | [standalone]

#57989

On 28/10/2013 7:50 PM, Wolfgang Maier wrote:
> imagine you have a flag set somewhere earlier in your code, e.g.,
>
> needs_processing = True
>
> then in a for loop you're processing the elements of an iterable, but the
> kind of processing depends on the flag, e.g.,:
>
> for elem in iterable:
>      if needs_processing:
>          pre_process(elem)  # reformat elem in place
>      print(elem)
>
> this checks the condition every time through the for loop, even though there
> is no chance for needs_processing to change inside the loop, which does not
> look very efficient.

There are two approaches I would consider using here:

1. Make pre_process a decorator, and outside of the loop do:

     def pre_process_decorator(fn):
         def pre_process(x):
             # functionality goes here
             return fn(x)
         return pre_process

     if needs_processing:
         print = pre_process_decorator(print)

     for elem in iterable:
         print(elem)

2. Replace the iterable with a generator if the condition is met:

     if needs_processing:
         iterable = (pre_process(x) for x in iterable)

     for elem in iterable:
         print(elem)

Personally, I find the 2nd approach clearer.

[toc] | [prev] | [next] | [standalone]

#58038

On Monday, October 28, 2013 6:50:19 AM UTC-3, Wolfgang Maier wrote:
> Dear all,
> 
> this is a recurring programming problem that I'm just not sure how to solve
> 
> optimally, so I thought I'd ask for your advice:
> 
> imagine you have a flag set somewhere earlier in your code, e.g.,
> 
> 
> 
> needs_processing = True
> 
> 
> 
> then in a for loop you're processing the elements of an iterable, but the
> 
> kind of processing depends on the flag, e.g.,:
> 
> 
> 
> for elem in iterable:
> 
>     if needs_processing:
> 
>         pre_process(elem)  # reformat elem in place
> 
>     print(elem)
> 
> 
> 
> this checks the condition every time through the for loop, even though there
> 
> is no chance for needs_processing to change inside the loop, which does not
> 
> look very efficient. Of course, you could rewrite the above as:
> 
> 
> 
> if needs_processing:
> 
>     for elem in iterable:
> 
>         pre_process(elem)  # reformat elem in place
> 
>         print(elem)
> 
> else:
> 
>     for elem in iterable:
> 
>         print(elem)
> 
> 
> 
> but this means unnecessary code-duplication.
> 
> 
> 
> You could also define functions (or class methods):
> 
> def pre_process_and_print (item):
> 
>     pre_process(item)
> 
>     print(item)
> 
> 
> 
> def raw_print (item):
> 
>     print(item)
> 
> 
> 
> then:
> 
> process = pre_process_and_print if needs_processing else raw_print
> 
> for elem in iterable:
> 
>     process(elem)
> 
> 
> 
> but while this works for the simple example here, it becomes complicated if
> 
> pre_process requires more information to do its job because then you will
> 
> have to start passing around (potentially lots of) arguments.
> 
> 
> 
> So my question is: is there an agreed-upon generally best way of dealing
> 
> with this?
> 
> 
> 
> Thanks for your help,
> 
> Wolfgang

Hi All, 
Trying to help you out, I wonder if something like the following code will help on what you need.

for elem in (needs_processing and iterable or []):
    pre_process(elem)  # reformat elem in place
    print(elem) 

It avoids code duplication and only process and iterates if the condition is True.
I hope it helps, otherwise, we could keep thinking alternatives.

Regards.
Mariano.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web