Re: Why does lzma hangs for a very long time when run in parallel using python's muptiprocessing module?

Path	csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.029
X-Spam-Evidence	'H': 0.94; 'S': 0.00; 'below)': 0.09; 'subject:Why': 0.09; 'subject:module': 0.09; 'subject:using': 0.09; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'subject: \n ': 0.16; 'subject:run': 0.16; 'subject:skip:m 10': 0.16; 'subject:when': 0.16; 'subject:python': 0.16; 'sat,': 0.16; 'wrote:': 0.18; 'bit': 0.19; 'trying': 0.19; 'fine': 0.24; '(see': 0.26; 'shown': 0.26; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; 'run': 0.32; 'subject:time': 0.33; "can't": 0.35; 'but': 0.35; 'received:google.com': 0.35; 'version': 0.36; 'subject:?': 0.36; 'confirmed': 0.38; 'to:addr:python-list': 0.38; 'does': 0.39; 'use.': 0.39; 'to:addr:python.org': 0.39; 'details.': 0.61; 'hang': 0.67; '26,': 0.68; 'below.': 0.71; 'subject:long': 0.84; 'sorry.': 0.91; 'subject:very': 0.91; '2013': 0.98
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Q7rBo4OMQFi5/8yMo1JnaWiYFB8ylPC3al9UkRseHng=; b=M/NqlycVMZg/fHAcNgNHAR96Gxu108PFmKihuAWjD4IaiNdX48xMWk/Y+k34gW/D98 6pM+wbE3ACt2m8P6/u0EfWSJ0pVf90rOp8dcO/w+uAi4TW3QB6jS9UDfYNpZY8V4ig+w 2vsHQrMzMkAQvT1fGHhxtoeEQ9XsvCCkcB3ERVxtzvyC2wKct8/CYwYZNW+2B9M8ykLs D0hVMVoLAigYUWMB411slpP+sAHij+PoVEIJK3jZfcNnDrDJFM9wi7dlhR2cT3xZqWRg LGhmSvwtcv7490L+lJnxK/DTNjF0hZoc+dzQnltzHAUlM6lMENwfUnb6JL3duLZayzEK T+HQ==
MIME-Version	1.0
X-Received	by 10.68.189.101 with SMTP id gh5mr9442167pbc.39.1382748492080; Fri, 25 Oct 2013 17:48:12 -0700 (PDT)
In-Reply-To	<CACxeJGrM7L3i323fshfm-NjEtXomwtNEbNepVtJhmv9EnMqixA@mail.gmail.com>
References	<CACxeJGrM7L3i323fshfm-NjEtXomwtNEbNepVtJhmv9EnMqixA@mail.gmail.com>
Date	Sat, 26 Oct 2013 11:48:12 +1100
Subject	Re: Why does lzma hangs for a very long time when run in parallel using python's muptiprocessing module?
From	Chris Angelico <rosuav@gmail.com>
To	python-list@python.org
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.1567.1382748495.18130.python-list@python.org> (permalink)
Lines	10
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1382748495 news.xs4all.nl 15915 [2001:888:2000:d::a6]:48340
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:57586

Show key headers only | View raw

On Sat, Oct 26, 2013 at 3:21 AM, cantor cantor <cantormath2@gmail.com> wrote:
> When trying to run lzma in parallel (see the code below) it hangs for a very
> long time. The non-parallel version of the code using map() works fine as
> shown in the code below.

Confirmed that your code does indeed hang as you describe, but I can't
help much with the details. I poked around with it a bit but without
finding much of use. Sorry. :(

ChrisA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Why does lzma hangs for a very long time when run in parallel using python's muptiprocessing module? Chris Angelico <rosuav@gmail.com> - 2013-10-26 11:48 +1100

csiph-web