Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.002 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'programmer': 0.03; 'board.': 0.05; 'interpreter': 0.05; 'that?': 0.05; 'context': 0.07; 'explicit': 0.07; 'interpreter.': 0.07; 'level,': 0.07; 'subject:help': 0.08; 'interpreted': 0.09; 'python': 0.11; 'thread': 0.14; 'times,': 0.14; '"every': 0.16; 'bytecode': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'so.': 0.16; 'wrote:': 0.18; 'code.': 0.18; 'seems': 0.21; 'separate': 0.22; "aren't": 0.24; 'gets': 0.27; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'message-id:@mail.gmail.com': 0.30; 'gives': 0.31; 'code': 0.31; 'usually': 0.31; 'consequence': 0.31; 'everyone': 0.33; 'level.': 0.33; 'sense': 0.34; 'core': 0.34; 'problem': 0.35; 'subject:with': 0.35; "can't": 0.35; 'something': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'really': 0.36; 'done,': 0.36; 'proposals': 0.36; 'tight': 0.36; 'done': 0.36; 'doing': 0.36; "didn't": 0.36; 'operating': 0.37; 'two': 0.37; 'level': 0.37; 'performance': 0.37; 'implement': 0.38; 'to:addr:python-list': 0.38; 'issue': 0.38; 'fact': 0.38; 'anything': 0.39; 'realize': 0.39; 'to:addr:python.org': 0.39; 'how': 0.40; 'easy': 0.60; 'catch': 0.60; 'lower': 0.61; 'making': 0.63; 'between': 0.67; 'cpu.': 0.84; 'usage.': 0.84; 'carlos': 0.91; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=fbRanmiDIuu1AHioF9QJ9UMLbunyG+Hl/EOUGr4ETCE=; b=pB1666aQHnBQmrj5daBuiw7M4Ski9E/f0INo4zg3UHGk2mmt6IXsaZWsKWZNwoQ1fx nmt0vDAz7h3eBMmz/MsCREH6oq3owApoSn/486kI7k1c2Y3z74TrIVRvUMGb/uHji4JW rIL5XSi0XG7dXXba1Q+fZwnil2NKpsBSkVrTU4KEp7fBGoQvNp1CCvMKhRQGMDvVuQnL iqkzhNqDWZ5AFMi9NMTLq13/zzjjrQ3NhfJ0fKkMR+9UZ7DTs8SaTCWx330Iw9aixaRg jDXgcy5xQaaW24Vh5l4wdu9v7wYuiKhuRZmP5hKgZNGp2h1eQlWGDH2Wr5QhoGtYPQ0O JbtA== MIME-Version: 1.0 X-Received: by 10.52.65.238 with SMTP id a14mr11205916vdt.24.1368923894797; Sat, 18 May 2013 17:38:14 -0700 (PDT) In-Reply-To: References: <7baacf5a-0c50-4935-ad5b-148c208d759b@googlegroups.com> <13lfp8lds6e2e41rtsnvqimcb6inu7p28o@invalid.netcom.com> Date: Sun, 19 May 2013 10:38:14 +1000 Subject: Re: Please help with Threading From: Chris Angelico To: python-list@python.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 31 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1368923899 news.xs4all.nl 15947 [2001:888:2000:d::a6]:60444 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:45539 On Sun, May 19, 2013 at 10:02 AM, Carlos Nepomuceno wrote: > I didn't know Python threads aren't preemptive. Seems to be something really old considering the state of the art on parallel execution on multi-cores. > > What's the catch on making Python threads preemptive? Are there any ongoing projects to make that? Preemption isn't really the issue here. On the C level, preemptive vs cooperative usually means the difference between a stalled thread locking everyone else out and not doing so. Preemption is done at a lower level than user code (eg the operating system or the CPU), meaning that user code can't retain control of the CPU. With interpreted code eg in CPython, it's easy to implement preemption in the interpreter. I don't know how it's actually done, but one easy implementation would be "every N bytecode instructions, context switch". It's still done at a lower level than user code (N bytecode instructions might all actually be a single tight loop that the programmer didn't realize was infinite), but it's not at the OS level. But none of that has anything to do with multiple core usage. The problem there is that shared data structures need to be accessed simultaneously, and in CPython, there's a Global Interpreter Lock to simplify that; but the consequence of the GIL is that no two threads can simultaneously execute user-level code. There have been GIL-removal proposals at various times, but the fact remains that a global lock makes a huge amount of sense and gives pretty good performance across the board. There's always multiprocessing when you need multiple CPU-bound threads; it's an explicit way to separate the shared data (what gets transferred) from local (what doesn't). ChrisA