Path: csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.005 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'scripts': 0.03; 'interpreter': 0.05; 'that?': 0.05; 'nested': 0.07; 'subject:help': 0.08; '(instead': 0.09; 'interpreted': 0.09; 'works.': 0.09; 'runs': 0.10; 'python': 0.11; 'thread': 0.14; 'conflicting': 0.16; 'dict': 0.16; 'dictionaries': 0.16; 'interpreter,': 0.16; 'iterating': 0.16; 'loops': 0.16; 'numpy': 0.16; 'processor,': 0.16; 'suggest?': 0.16; 'thread,': 0.16; 'threads.': 0.16; 'url:dabeaz': 0.16; 'elements': 0.16; 'prevent': 0.16; 'sat,': 0.16; 'library': 0.18; 'trying': 0.19; 'starts': 0.20; 'seems': 0.21; 'code,': 0.22; 'to:name:python- list@python.org': 0.22; 'creating': 0.23; "aren't": 0.24; 'instead.': 0.24; 'received:65.55.116': 0.24; 'url:home': 0.24; 'fine': 0.24; 'script': 0.25; 'pass': 0.26; 'header:In-Reply- To:1': 0.27; 'skip:- 40': 0.29; 'array': 0.29; 'thus': 0.29; "doesn't": 0.30; 'url:mailman': 0.30; 'went': 0.31; 'code': 0.31; 'complete,': 0.31; 'long.': 0.31; 'overhead': 0.31; 'lists': 0.32; 'this.': 0.32; 'run': 0.32; 'another': 0.32; 'up.': 0.33; 'url:python': 0.33; 'mac': 0.33; 'actual': 0.34; 'date:': 0.34; 'core': 0.34; 'subject:with': 0.35; 'info': 0.35; 'common': 0.35; 'created': 0.35; 'something': 0.35; 'but': 0.35; 'there': 0.35; 'really': 0.36; 'module.': 0.36; 'url:listinfo': 0.36; 'doing': 0.36; 'next': 0.36; "didn't": 0.36; 'url:org': 0.36; 'list': 0.37; 'email addr:python.org': 0.37; 'level': 0.37; 'list.': 0.37; 'being': 0.38; 'to:addr:python-list': 0.38; 'resource': 0.38; 'that,': 0.38; 'subject:': 0.39; 'itself': 0.39; 'to:addr:python.org': 0.39; 'either': 0.39; 'release': 0.40; 'url:mail': 0.40; 'even': 0.60; 'catch': 0.60; 'devices': 0.61; 'first': 0.61; "you'll": 0.62; 'making': 0.63; 're:': 0.63; 'such': 0.63; 'more': 0.64; 'email name:python-list': 0.65; 'between': 0.67; 'url:pdf': 0.68; 'sharing': 0.69; 'intelligent': 0.74; 'goal': 0.75; 'gain': 0.79; '(while': 0.84; '80%': 0.84; 'multicore': 0.84; 'odds': 0.84; 'drops': 0.91; '2013': 0.98 X-TMN: [0j6lY7uaXGOfRyGSIoSOKA5J1uRfMYk1] X-Originating-Email: [carlosnepomuceno@outlook.com] From: Carlos Nepomuceno To: "python-list@python.org" Subject: RE: Please help with Threading Date: Sun, 19 May 2013 03:02:29 +0300 Importance: Normal In-Reply-To: <13lfp8lds6e2e41rtsnvqimcb6inu7p28o@invalid.netcom.com> References: <7baacf5a-0c50-4935-ad5b-148c208d759b@googlegroups.com>, <13lfp8lds6e2e41rtsnvqimcb6inu7p28o@invalid.netcom.com> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 19 May 2013 00:02:30.0164 (UTC) FILETIME=[27AEB540:01CE5424] X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 76 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1368921819 news.xs4all.nl 15869 [2001:888:2000:d::a6]:52201 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:45537 ----------------------------------------=0A= > To: python-list@python.org=0A= > From: wlfraed@ix.netcom.com=0A= > Subject: Re: Please help with Threading=0A= > Date: Sat=2C 18 May 2013 15:28:56 -0400=0A= >=0A= > On Sat=2C 18 May 2013 01:58:13 -0700 (PDT)=2C Jurgens de Bruin=0A= > declaimed the following in=0A= > gmane.comp.python.general:=0A= >=0A= >> This is my first script where I want to use the python threading module.= I have a large dataset which is a list of dict this can be as much as 200 = dictionaries in the list. The final goal is a histogram for each dict 16 hi= stograms on a page ( 4x4 ) - this already works.=0A= >> What I currently do is a create a nested list [ [ {} ]=2C [ {} ] ] each = inner list contains 16 dictionaries=2C thus each inner list is a single pag= e of 16 histograms. Iterating over the outer-list and creating the graphs t= akes to long. So I would like multiple inner-list to be processes simultane= ously and creating the graphs in "parallel".=0A= >> I am trying to use the python threading for this. I create 4 threads loo= p over the outer-list and send a inner-list to the thread. This seems to wo= rk if my nested lists only contains 2 elements - thus less elements than th= reads. Currently the scripts runs and then seems to get hung up. I monitor = the resource on my mac and python starts off good using 80% and when the 4-= thread is created the CPU usages drops to 0%.=0A= >>=0A= >=0A= > The odds are good that this is just going to run slower...=0A= =0A= Just been told that GIL doesn't make things slower=2C but as I didn't know = that such a thing even existed I went out looking for more info and found t= hat document: http://www.dabeaz.com/python/UnderstandingGIL.pdf=0A= =0A= Is it current? I didn't know Python threads aren't preemptive. Seems to be = something really old considering the state of the art on parallel execution= on multi-cores.=0A= =0A= What's the catch on making Python threads preemptive? Are there any ongoing= projects to make that?=0A= =0A= > One: The common Python implementation uses a global interpreter lock=0A= > to prevent interpreted code from interfering with itself in multiple=0A= > threads. So "number cruncher" applications don't gain any speed from=0A= > being partitioned into thread -- even on a multicore processor=2C only on= e=0A= > thread can have the GIL at a time. On top of that=2C you have the overhea= d=0A= > of the interpreter switching between threads (GIL release on one thread= =2C=0A= > GIL acquire for the next thread).=0A= >=0A= > Python threads work fine if the threads either rely on intelligent=0A= > DLLs for number crunching (instead of doing nested Python loops to=0A= > process a numeric array you pass it to something like NumPy which=0A= > releases the GIL while crunching a copy of the array) or they do lots of= =0A= > I/O and have to wait for I/O devices (while one thread is waiting for=0A= > the write/read operation to complete=2C another thread can do some number= =0A= > crunching).=0A= >=0A= > If you really need to do this type of number crunching in Python=0A= > level code=2C you'll want to look into the multiprocessing library=0A= > instead. That will create actual OS processes (each with a copy of the=0A= > interpreter=2C and not sharing memory) and each of those can run on a cor= e=0A= > without conflicting on the GIL.=0A= =0A= Which library do you suggest?=0A= =0A= > --=0A= > Wulfraed Dennis Lee Bieber AF6VN=0A= > wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/=0A= >=0A= > --=0A= > http://mail.python.org/mailman/listinfo/python-list =