Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: kushal.kumaran@gmail.com
In-Reply-To: <CAPxp-0RsHONTwJVW4zAdVrAyrryhC=HVzbotZB8+NOO9fiSAdQ@mail.gmail.com>
References: <CAPxp-0RsHONTwJVW4zAdVrAyrryhC=HVzbotZB8+NOO9fiSAdQ@mail.gmail.com>
From: Kushal Kumaran <kushal.kumaran+python@gmail.com>
Date: Tue, 18 Sep 2012 14:01:55 +0530
Subject: Re: How to queue functions
To: python-list@python.org
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.862.1347957139.27098.python-list@python.org>
Lines: 64
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:29416

On Tue, Sep 18, 2012 at 10:26 AM, Dhananjay <dhananjay.c.joshi@gmail.com> wrote:
> Dear all,
>
> I am trying to use multiprocessing module.
> I have 5 functions and 2000 input files.
>
> First, I want to make sure that these 5 functions execute one after the
> other.
> Is there any way that I could queue these 5 functions within the same script
> ?
>
>
> Next, as there are 2000 input files.
> I could queue them by queue.put() and get back to run one by one using
> queue.get() as follows:
>
> for file in files:
>         if '.dat.gz' in file:
>             q.put(file)
>
> while True:
>         item = q.get()
>         x1 = f1(item)
>         x2 = f2(x1)
>         x3 = f3(x2)
>         x4 = f4(x3)
>         final_output = f5(x4)
>
>
> However, how can I input them on my 8 core machine, so that at a time 8
> files will be processed (to the set of 5 functions; each function one after
> the other) ?
>

The multiprocessing.Pool class seems to be what you need.
Documentation at
http://docs.python.org/py3k/library/multiprocessing.html#using-a-pool-of-workers

Example:

#!/usr/bin/env python3
import multiprocessing

def file_handler(filename):
    # do processing on filename, return the final_output
    print('working on {}'.format(filename))
    return 'processed-{}'.format(filename)

def main():
    p = multiprocessing.Pool(8)
    files = [ 'a', 'b', 'c' ]
    result = p.map(file_handler, files)
    print(result)

if __name__ == '__main__':
    main()

If you want, you can also implement everything using
multiprocessing.Process and multiprocessing.Queue, but using pools
should be simpler.

-- 
regards,
kushal