Path: csiph.com!usenet.pasdenom.info!news.etla.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <ethan@stoneleaf.us>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.016
X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'skip:[ 20': 0.04; 'output': 0.05; 'from:addr:ethan': 0.09; 'from:addr:stoneleaf.us': 0.09; 'from:name:ethan furman': 0.09; 'input,': 0.09; 'lines:': 0.09; 'message-id:@stoneleaf.us': 0.09; 'params': 0.09; '~ethan~': 0.09; 'thread': 0.14; 'subject:requests': 0.16; 'wrote:': 0.18; 'input': 0.22; 'python?': 0.22; 'this?': 0.23; 'header:User- Agent:1': 0.23; 'skip:` 20': 0.24; 'this:': 0.26; 'header:In- Reply-To:1': 0.27; 'function': 0.29; 'am,': 0.29; 'lines': 0.31; 'anyone': 0.31; "i'd": 0.34; 'could': 0.34; 'problem': 0.35; 'basic': 0.35; 'hi,': 0.36; 'two': 0.37; 'to:addr:python-list': 0.38; 'does': 0.39; 'to:addr:python.org': 0.39; 'hope': 0.61; 'entire': 0.61; 'received:173': 0.61; 'useful.': 0.68; 'limit': 0.70; 'miss': 0.74; '(incl.': 0.84; 'pools': 0.84; 'received:67.18.68': 0.84; 'received:67.18.68.2': 0.84; 'received:gateway14.websitewelcome.com': 0.84; 'ugly,': 0.84
Date: Thu, 03 Oct 2013 10:01:02 -0700
From: Ethan Furman <ethan@stoneleaf.us>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: feature requests
References: <6782f295-1885-4114-aea8-d785480f3489@googlegroups.com>
In-Reply-To: <6782f295-1885-4114-aea8-d785480f3489@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - gator3304.hostgator.com
X-AntiAbuse: Original Domain - python.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - stoneleaf.us
X-BWhitelist: no
X-Source: 
X-Source-Args: 
X-Source-Dir: 
X-Source-Sender: ([173.12.184.233]) [173.12.184.233]:34536
X-Source-Auth: ethan+stoneleaf.us
X-Email-Count: 1
X-Source-Cap: dG9idWs7dG9idWs7Z2F0b3IzMzA0Lmhvc3RnYXRvci5jb20=
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.687.1380822918.18130.python-list@python.org>
Lines: 31
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1380822918 news.xs4all.nl 15901 [2001:888:2000:d::a6]:43721
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:55429

On 10/03/2013 09:12 AM, macker wrote:
> Hi, hope this is the right group for this:
>
> I miss two basic (IMO) features in parallel processing:
>
> 1. make `threading.Thread.start()` return `self`
>
> I'd like to be able to `workers = [Thread(params).start() for params in whatever]`. Right now, it's 5 ugly, menial lines:
>
>          workers = []
>          for params in whatever:
>              thread = threading.Thread(params)
>              thread.start()
>              workers.append(thread)

Ugly, menial lines are a clue that a function to hide it could be useful.


> 2. make multiprocessing pools (incl. ThreadPool) limit the size of their internal queues
>
> As it is now, the queue will greedily consume its entire input, and if the input is large and the pool workers are slow in consuming it, this blows up RAM. I'd like to be able to `pool = Pool(4, max_qsize=1000)`. Same with the output queue (finished tasks).

Have you verified that this is a problem in Python?


> Or does anyone know of a way to achieve this?

You could try subclassing.

--
~Ethan~