Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder3.xlned.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <513DF0BC.1000902@uci.edu>
References: <1522066129.3339778.1363000462968.JavaMail.root@sequans.com> <513DF0BC.1000902@uci.edu>
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
Date: Mon, 11 Mar 2013 15:58:54 +0000
Subject: Re: Advice regarding multiprocessing module
To: Abhinav M Kulkarni <amkulkar@uci.edu>
Content-Type: text/plain; charset=ISO-8859-1
Cc: Python List <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3193.1363017562.2939.python-list@python.org>
Lines: 63
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:41073

On 11 March 2013 14:57, Abhinav M Kulkarni <amkulkar@uci.edu> wrote:
> Hi Jean,
>
> Below is the code where I am creating multiple processes:
>
> if __name__ == '__main__':
>     # List all files in the games directory
>     files = list_sgf_files()
>
>     # Read board configurations
>     (intermediateBoards, finalizedBoards) = read_boards(files)
>
>     # Initialize parameters
>     param = Param()
>
>     # Run maxItr iterations of gradient descent
>     for itr in range(maxItr):
>         # Each process analyzes one single data point
>         # They dump their gradient calculations in queue q
>         # Queue in Python is process safe
>         start_time = time.time()
>         q = Queue()
>         jobs = []
>         # Create a process for each game board
>         for i in range(len(files)):
>             p = Process(target=TrainGoCRFIsingGibbs, args=(intermediateBoards[i], finalizedBoards[i], param, q))

Use a multiprocessing.Pool for this, rather than creating one process
for each job. e.g.:

p = Pool(4) # 1 process for each core
results = []
for ib, fb in zip(intermediateBoards, finalizedBoards):
    results.append(p.apply_async(TrainGoCRFIsingGibbs, args=(ib, fb, param, q)))
p.close()
p.join()

# To retrieve the return values
for r in results:
    print(r.get())

This will distribute your jobs over a fixed number of processes. You
avoid the overhead of creating and killing processes and the process
switching that occurs when you have more processes than cores.

>             p.start()
>             jobs.append(p)
>         # Blocking wait for each process to finish
>         for p in jobs:
>             p.join()
>         elapsed_time = time.time() - start_time
>         print 'Iteration: ', itr, '\tElapsed time: ', elapsed_time
>
> As you recommended, I'll use the profiler to see which part of the code is
> slow.

Do this without using multiprocessing first. Loosely you can hope that
multiprocessing would give you a factor of 4 speedup but no more. You
haven't reported a comparison of times with/without multiprocessing so
it's not clear that that is the issue.


Oscar