Re: feature requests

Path	csiph.com!usenet.pasdenom.info!news.etla.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<ethan@stoneleaf.us>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.016
X-Spam-Evidence	'H': 0.97; 'S': 0.00; 'skip:[ 20': 0.04; 'output': 0.05; 'from:addr:ethan': 0.09; 'from:addr:stoneleaf.us': 0.09; 'from:name:ethan furman': 0.09; 'input,': 0.09; 'lines:': 0.09; 'message-id:@stoneleaf.us': 0.09; 'params': 0.09; '~ethan~': 0.09; 'thread': 0.14; 'subject:requests': 0.16; 'wrote:': 0.18; 'input': 0.22; 'python?': 0.22; 'this?': 0.23; 'header:User- Agent:1': 0.23; 'skip:` 20': 0.24; 'this:': 0.26; 'header:In- Reply-To:1': 0.27; 'function': 0.29; 'am,': 0.29; 'lines': 0.31; 'anyone': 0.31; "i'd": 0.34; 'could': 0.34; 'problem': 0.35; 'basic': 0.35; 'hi,': 0.36; 'two': 0.37; 'to:addr:python-list': 0.38; 'does': 0.39; 'to:addr:python.org': 0.39; 'hope': 0.61; 'entire': 0.61; 'received:173': 0.61; 'useful.': 0.68; 'limit': 0.70; 'miss': 0.74; '(incl.': 0.84; 'pools': 0.84; 'received:67.18.68': 0.84; 'received:67.18.68.2': 0.84; 'received:gateway14.websitewelcome.com': 0.84; 'ugly,': 0.84
Date	Thu, 03 Oct 2013 10:01:02 -0700
From	Ethan Furman <ethan@stoneleaf.us>
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1
MIME-Version	1.0
To	python-list@python.org
Subject	Re: feature requests
References	<6782f295-1885-4114-aea8-d785480f3489@googlegroups.com>
In-Reply-To	<6782f295-1885-4114-aea8-d785480f3489@googlegroups.com>
Content-Type	text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding	7bit
X-AntiAbuse	This header was added to track abuse, please include it with any abuse report
X-AntiAbuse	Primary Hostname - gator3304.hostgator.com
X-AntiAbuse	Original Domain - python.org
X-AntiAbuse	Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse	Sender Address Domain - stoneleaf.us
X-BWhitelist	no
X-Source
X-Source-Args
X-Source-Dir
X-Source-Sender	([173.12.184.233]) [173.12.184.233]:34536
X-Source-Auth	ethan+stoneleaf.us
X-Email-Count	1
X-Source-Cap	dG9idWs7dG9idWs7Z2F0b3IzMzA0Lmhvc3RnYXRvci5jb20=
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.687.1380822918.18130.python-list@python.org> (permalink)
Lines	31
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1380822918 news.xs4all.nl 15901 [2001:888:2000:d::a6]:43721
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:55429

Show key headers only | View raw

On 10/03/2013 09:12 AM, macker wrote:
> Hi, hope this is the right group for this:
>
> I miss two basic (IMO) features in parallel processing:
>
> 1. make `threading.Thread.start()` return `self`
>
> I'd like to be able to `workers = [Thread(params).start() for params in whatever]`. Right now, it's 5 ugly, menial lines:
>
>          workers = []
>          for params in whatever:
>              thread = threading.Thread(params)
>              thread.start()
>              workers.append(thread)

Ugly, menial lines are a clue that a function to hide it could be useful.


> 2. make multiprocessing pools (incl. ThreadPool) limit the size of their internal queues
>
> As it is now, the queue will greedily consume its entire input, and if the input is large and the pool workers are slow in consuming it, this blows up RAM. I'd like to be able to `pool = Pool(4, max_qsize=1000)`. Same with the output queue (finished tasks).

Have you verified that this is a problem in Python?


> Or does anyone know of a way to achieve this?

You could try subclassing.

--
~Ethan~

Thread

feature requests macker <tester.testerus@gmail.com> - 2013-10-03 09:12 -0700
  Re: feature requests Chris Angelico <rosuav@gmail.com> - 2013-10-04 02:21 +1000
  Re: feature requests Tim Chase <python.list@tim.thechases.com> - 2013-10-03 11:42 -0500
  Re: feature requests Chris Angelico <rosuav@gmail.com> - 2013-10-04 02:42 +1000
  Re: feature requests Ethan Furman <ethan@stoneleaf.us> - 2013-10-03 10:01 -0700
    Re: feature requests macker <tester.testerus@gmail.com> - 2013-10-05 05:49 -0700
      Re: feature requests Ethan Furman <ethan@stoneleaf.us> - 2013-10-05 08:58 -0700
      Re: feature requests Terry Reedy <tjreedy@udel.edu> - 2013-10-05 17:56 -0400

csiph-web