Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #7015 > unrolled thread

Generator Frustration

Back to article view | Back to comp.lang.python

Contents

  Generator Frustration "TommyVee" <xxxxxxxx@xxxxxx.xxx> - 2011-06-04 14:27 -0400
    Re: Generator Frustration Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-06-05 00:56 +0000
      Re: Generator Frustration Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-06-05 13:43 +1200
        Re: Generator Frustration Jack Diederich <jackdied@gmail.com> - 2011-06-04 22:06 -0400
        Re: Generator Frustration "TommyVee" <xxxxxxxx@xxxxxx.xxx> - 2011-06-05 20:11 -0400
    Re: Generator Frustration Jan Decaluwe <jan@jandecaluwe.com> - 2011-06-05 11:52 +0200
    Re: Generator Frustration Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-06-06 11:07 +0200
      Re: Generator Frustration "TommyVee" <xxxxxxxx@xxxxxx.xxx> - 2011-06-07 20:41 -0400

#7015 — Generator Frustration

I'm using the SimPy package to run simulations. Anyone who's used this 
package knows that the way it simulates process concurrency is through the 
clever use of yield statements. Some of the code in my programs is very 
complex and contains several repeating sequences of yield statements.  I 
want to combine these sequences into common functions.  The problem of 
course, is that once a yield gets put into a function, the function is now a 
generator and its behavior changes.  Is there any elegant way to do this?  I 
suppose I can do things like ping-pong yield statements, but that solutions 
seems even uglier than having a very flat, single main routine with 
repeating sequences.

Thanks in advance. 

[toc] | [next] | [standalone]

#7027

On Sat, 04 Jun 2011 14:27:32 -0400, TommyVee wrote:

> I'm using the SimPy package to run simulations. Anyone who's used this
> package knows that the way it simulates process concurrency is through
> the clever use of yield statements. Some of the code in my programs is
> very complex and contains several repeating sequences of yield
> statements.  I want to combine these sequences into common functions. 
> The problem of course, is that once a yield gets put into a function,
> the function is now a generator and its behavior changes.  Is there any
> elegant way to do this?


I don't quite understand the nature of your problem, but it seems to me 
that it is easily solved by simply not using yield in the common 
functions. Instead of:


def main():
    for x in sequence:
        if a:
            y = a+b+c+d+e
            yield y
        elif b:
            y = a+b+c+d+f
            yield y
        else:
            y = a+b+c+d
            yield y+1

(where each of the lines y = ... is meant to be a big block of mostly 
common code), factor out the similar parts into one or more external 
functions:

def f(a, b, c, d):
    return a+b+c+d  # big block of common code

def main():
    for x in sequence:
        if a:
            y = f(a, b, c, d)  # call the common part
            y += e
            yield y
        elif b:
            y = f(a, b, c, d)
            y += f
            yield y
        else:
            y = f(a, b, c, d)
            y += 1
            yield y


If this is not what you're talking about, an example may help.


> I suppose I can do things like ping-pong yield statements, 

I have no idea what you think that means, but another alternative is to 
loop over the generator output and re-yield it:

        for x in f():
            yield x

A nice piece of syntax that has been proposed for Python is "yield from", 
which will do the same thing, but you can't use that yet.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#7029

Steven D'Aprano wrote:

> A nice piece of syntax that has been proposed for Python is "yield from", 
> which will do the same thing, but you can't use that yet.

Unless you're impatient enough to compile your own
Python with my patch applied:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#7031

On Sat, Jun 4, 2011 at 9:43 PM, Gregory Ewing
<greg.ewing@canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>
>> A nice piece of syntax that has been proposed for Python is "yield from",
>> which will do the same thing, but you can't use that yet.

You can also patch the library to always return lists instead of generators.

def seriously_I_just_want_lists(func):
  def replacement(*args, **kwargs):
    return list(func(*args, **kwargs))
  return replacement

import otherlib
otherlib.somefunc = seriously_I_just_want_lists(otherlib.somefunc)

But I'd avoid it where you can.  While the idea of a list is simpler
than the idea of an iterable (and easier to get your head around)
iterables promise less and that makes them more flexible and easier to
composite.

-Jack

[toc] | [prev] | [next] | [standalone]

#7062

"Gregory Ewing"  wrote in message news:95059eFuroU1@mid.individual.net...

Steven D'Aprano wrote:

> A nice piece of syntax that has been proposed for Python is "yield from", 
> which will do the same thing, but you can't use that yet.

Unless you're impatient enough to compile your own
Python with my patch applied:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html

-- 
Greg

Sounds like the "yield from" is the about this best solution.  Thanks for 
all your help. 

[toc] | [prev] | [next] | [standalone]

#7040

On 06/04/2011 08:27 PM, TommyVee wrote:
> I'm using the SimPy package to run simulations. Anyone who's used
> this package knows that the way it simulates process concurrency is
> through the clever use of yield statements. Some of the code in my
> programs is very complex and contains several repeating sequences of
> yield statements. I want to combine these sequences into common
> functions. The problem of course, is that once a yield gets put into
> a function, the function is now a generator and its behavior changes.
> Is there any elegant way to do this? I suppose I can do things like
> ping-pong yield statements, but that solutions seems even uglier than
> having a very flat, single main routine with repeating sequences.

In MyHDL, this is supported by the possibility to yield generators,
which are then handled by the simulation engine.

Jan


-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
     Python as a HDL: http://www.myhdl.org
     VHDL development, the modern way: http://www.sigasi.com
     Analog design automation: http://www.mephisto-da.com
     World-class digital design: http://www.easics.com

[toc] | [prev] | [next] | [standalone]

#7081

Am 04.06.2011 20:27 schrieb TommyVee:
> I'm using the SimPy package to run simulations. Anyone who's used this
> package knows that the way it simulates process concurrency is through
> the clever use of yield statements. Some of the code in my programs is
> very complex and contains several repeating sequences of yield
> statements. I want to combine these sequences into common functions.

Which are then generators.

 > The problem of course, is that once a yield gets put into a function,
 > the function is now a generator and its behavior changes.

Isn't your "main" function a generator as well?


> Is there  any elegant way to do this? I suppose I can do things like
 > ping-pong yield statements, but that solutions seems even uglier than
 > having a very flat, single main routine with repeating sequences.

I'm not sure if I got it right, but I think you could emulate this 
"yield from" with a decorator:

def subgen1(): yield 1; yield 2;
def subgen2(): yield 1; yield 2;

Instead of doing now

def allgen():
     for i in subgen1(): yield i
     for i in subgen2(): yield i

you as well could do:

def yield_from(f):
     def wrapper(*a, **k):
         for sub in f(*a, **k):
             for i in sub:
                 yield i
     return wrapper

@yield_from
def allgen():
     yield subgen1()
     yield subgen2()

(Untested.)


Thomas

[toc] | [prev] | [next] | [standalone]

#7203

"Thomas Rachel"  wrote in message news:isi5dk$8h1$1@r03.glglgl.eu...

Am 04.06.2011 20:27 schrieb TommyVee:
> I'm using the SimPy package to run simulations. Anyone who's used this
> package knows that the way it simulates process concurrency is through
> the clever use of yield statements. Some of the code in my programs is
> very complex and contains several repeating sequences of yield
> statements. I want to combine these sequences into common functions.

Which are then generators.

> The problem of course, is that once a yield gets put into a function,
> the function is now a generator and its behavior changes.

Isn't your "main" function a generator as well?


> Is there  any elegant way to do this? I suppose I can do things like
> ping-pong yield statements, but that solutions seems even uglier than
> having a very flat, single main routine with repeating sequences.

I'm not sure if I got it right, but I think you could emulate this
"yield from" with a decorator:

def subgen1(): yield 1; yield 2;
def subgen2(): yield 1; yield 2;

Instead of doing now

def allgen():
     for i in subgen1(): yield i
     for i in subgen2(): yield i

you as well could do:

def yield_from(f):
     def wrapper(*a, **k):
         for sub in f(*a, **k):
             for i in sub:
                 yield i
     return wrapper

@yield_from
def allgen():
     yield subgen1()
     yield subgen2()

(Untested.)


Thomas

Yes, the main function is a generator.  Give me a day or two to absorb your 
code.  But in the meantime, perhaps a quick explanation of the SimPy package 
is in order.  Also, here's the link if you're interested:

http://simpy.sourceforge.net/simpy_overview.htm

The way the package works is that you instantiate what I'll call an "actor" 
class, inheriting from one of the SimPy base classes.  Once you instantiate 
the actor class(es), it it "registered" in SimPy.  Then when you start the 
simulation, SimPy's main routine will begin to "dispatch" each of the actor 
classes by driving what they call a "PEM" method in the actor class.  Within 
that method, you simulate the activity of the actor.  The way that you tell 
SimPy that you are "busy", or that you want to request or release a 
resource, is with a yield statement.  Here's a little example:

from SimPy.Simulation import *

svc1 = Resource(capacity=1)
svc2 = Resource(capacity=1)

class Customer(Process):
    def PEM(self):

        print now(), "I am starting..."

        print now(), "I am requesting service1"
        yield request, self, svc1

        print now(), "I got service 1, now I'm going ask it to do 10 ticks 
worth of work"
        yield hold, self, 10

        print now(), "Service 1's work is done, I am now releasing him"
        yield release, self, svc1

        print now(), "Now I am requesting service2"
        yield request, self, svc2

        print now(), "I got service 2, now I'm going ask him to do 5 ticks 
worth of work"
        yield hold, self, 5

        print now(), "Service 2's work is done, I am now releasing him"
        yield release, self, svc2

        print now(), "I am ending..."

initialize()

# Create a customer and "register" (activate) him to SimPy
c = Customer()
activate(c, c.PEM())

# Pass control to the SimPy main dispatch routine, and run the simulation 
for 100 ticks
simulate(until=100)

Note that when you do the yields, you actually send back information to the 
SimPy main routine.  For example, when you do a "yield hold, self, 10", 
control will be yielded to SimPy, he will simulate the passage of 10 time 
ticks, and redispatch you (do a "next" on your PEM).  Similar deal with the 
"yield request, self, svc1".  When you yield to SimPy, he will attempt to 
obtain the resource for you, and if it is already taken by another actor, 
you will wait in a queue until released.  At that time, SimPy will then 
redispatch you.  Obviously, this is a trivial example, since it only 
involves the creation of a single actor.  In a real simulation, there could 
be hundreds of these things running "concurrently", all vying for resources, 
holding them for varying amounts of time, etc. SimPy also uses yield 
statements to simulate other things too, like waiting for a signal from 
another actor.

In the example you see above, note that I am "repeating" a generic sequence 
twice, e.g. "yield request, self, svcX", followed by "yield hold, self, 
time", followed by "yield release, self, svcX".  In a workflow example, you 
may have literally dozens of services that you may hit in serial or 
parallel.  Because of the generator "restriction",  you'd have to code these 
three statements repeatedly in the PEM routine.  It would be nice to just 
have a class method which did:

def UseService(self, svcName, svcTime):
    yield request, self, svcName
    yield hold, self, svcTime
    yield release, self, svcName

Then you can just call it in your main routine, passing the desired 
parameters (e.g., "UseService(svc1, 10)").

Anyway, hopefully you'll see that there's a reason for my original question. 
Let me absorb what you sent and see if it makes sense.

Thanks, Tom 

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web