Path: csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder2.enfer-du-nord.net!newsfeed.eweka.nl!eweka.nl!feeder3.eweka.nl!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <davea@davea.name>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.005
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'python.': 0.02; 'ideally': 0.04; 'linux,': 0.05; '(instead': 0.09; 'subject:module': 0.09; 'subject:skip:m 10': 0.09; 'thread': 0.11; 'do,': 0.15; 'processor,': 0.16; 'slow,': 0.16; 'wrote:': 0.17; 'instance': 0.17; 'intel': 0.17; 'module': 0.19; 'seems': 0.23; 'specifically': 0.24; 'header:In-Reply-To:1': 0.25; 'header:User- Agent:1': 0.26; 'creating': 0.26; 'am,': 0.27; 'code': 0.31; 'point': 0.31; 'subject:regarding': 0.33; 'to:addr:python-list': 0.33; 'know.': 0.33; 'another': 0.33; 'but': 0.36; 'data.': 0.36; 'data': 0.37; 'subject:: ': 0.38; 'instead': 0.39; 'to:addr:python.org': 0.39; 'received:192': 0.39; 'received:192.168': 0.40; 'end': 0.40; 'time,': 0.62; 'laptop': 0.66; 'received:74.208': 0.71; '01:57': 0.84; 'received:74.208.4.194': 0.84; 'processes,': 0.93
Date: Mon, 11 Mar 2013 07:27:28 -0400
From: Dave Angel <davea@davea.name>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130221 Thunderbird/17.0.3
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Advice regarding multiprocessing module
References: <513D6FEB.9040706@uci.edu> <513D725D.307@uci.edu>
In-Reply-To: <513D725D.307@uci.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V02:K0:AHx8FK8Mzs9viXkC6STUbDVWaztAf8/hEwTxchGuXWr +zTI0N4qfTaIuZEk1W1NsuL5LtMayV0fw2FmKB5Os4l8IwZxdo 7Ig/eXkUXOB8OgoqVFhVHUvFxXaICR2QjDATsKMmzR1Z9yAF9U Wn1JrtxAKkzpoFVsA7lok0ClsbeeOSJXMaADnVOqCL67tXooA6 M/EieTEz0Jx2j/T6wwHaedMXgYR5n9MHseTERdI8yGy2ZwgZTf cF3A8GJX+KCF+N46/NomX65PGYMOITcz/gFOTow4Akr/mkFM3Y Yozx+dDxkowubXn8rFQVbPtgv2F0K81zCb8PQsZDfQhVb4WQA= =
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.3181.1363001273.2939.python-list@python.org>
Lines: 25
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1363001273 news.xs4all.nl 6922 [2001:888:2000:d::a6]:58690
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:41061

On 03/11/2013 01:57 AM, Abhinav M Kulkarni wrote:
>
>
>   <SNIP>
>   * My laptop has quad-core Intel i5 processor, so I thought using
>     multiprocessing module I can parallelize my code (basically
>     calculate gradient in parallel on multiple cores simultaneously).
    <SNIP>
>   * As a result I end up creating a process for each data point (instead
>     of a thread that I would ideally like to do, so as to avoid process
>     creation overhead).

Seems you only need 4 processes, as you have 4 cores.  Instead of 
creating a new one each time, reuse the same 4 processes, letting each 
do a quarter of the data.

It's not the process creation that's particularly slow, but all the 
initialization of starting another instance of Python.  If you're on 
Linux, you might be able to speed that up by using fork, but I don't 
specifically know.



-- 
DaveA