Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <drsalists@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'algorithm': 0.04; 'value,': 0.04; 'cpython': 0.05; 'subject:Python': 0.06; 'float': 0.07; 'pypy': 0.07; 'dan': 0.09; 'references,': 0.09; 'cc:addr :python-list': 0.11; 'python': 0.11; '(also': 0.16; '2*i': 0.16; 'cc:name:python list': 0.16; 'elements).': 0.16; 'former,': 0.16; 'heap': 0.16; 'heap,': 0.16; 'heapq': 0.16; 'latter,': 0.16; 'pypy?': 0.16; 'sorting': 0.16; 'surprises': 0.16; 'url:html)': 0.16; 'usage,': 0.16; 'worst': 0.16; 'elements': 0.16; 'subject: ?': 0.16; 'wrote:': 0.18; 'basically': 0.19; "hasn't": 0.19; 'code,': 0.22; 'memory': 0.22; 'aug': 0.22; 'putting': 0.22; 'cc:addr:python.org': 0.22; 'mon,': 0.24; "haven't": 0.24; 'cc:2**0': 0.24; 'sort': 0.25; 'compare': 0.26; 'values': 0.27; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'array': 0.29; '(like': 0.30; 'compared': 0.30; 'fastest': 0.30; 'message- id:@mail.gmail.com': 0.30; 'code': 0.31; 'node': 0.31; 'probably': 0.32; 'another': 0.32; 'guess': 0.33; 'problem': 0.35; 'good.': 0.35; 'test': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'really': 0.36; 'doing': 0.36; 'url:org': 0.36; 'should': 0.36; 'list': 0.37; 'pm,': 0.38; 'anything': 0.39; 'does': 0.39; 'how': 0.40; 'august': 0.61; "you're": 0.61; 'making': 0.63; 'url:index': 0.63; 'high': 0.63; 'more': 0.64; 'to:addr:gmail.com': 0.65; 'believe': 0.68; 'beat': 0.68; 'default': 0.69; '3.4': 0.84; 'flexibility,': 0.84; 'partial': 0.84; 'hand,': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6jMsff9hTLENsTirVZcXh/MxEX96a0s2dDcK7VGLB3o=; b=Teuy9BY1lMlfsN8aj51K0AmZcYEBvMwPE2wHyOHWSMEE/YVRAKZbak2aho9+/RmKbU RJm5fMPxPoGdrD0D2ItPB3I9n47qEHSyOp0fJjL1zQD06jmHZv/zj1mllxHS2GMH4imn ost3N27X+Tvs49h3McW8R29qrdyXDAYNLjEJUGRLC3H9tB2z9FM5cS9sIa4iv2HQ0jVe KTMcNFo6v5Yuv4L4OP1bDIsvrWsGAXtiKC0OXaq2XqP+IwzjtxZHGJyk2kug+vUZdgD0 Pm5R0W+durJGYCvTd87D4QLzWyXK6J8d0OOM0++btbmT7xAK4oJLmlaBGdhOeLAdKkT9 Yw2w==
MIME-Version: 1.0
X-Received: by 10.152.28.230 with SMTP id e6mr39365670lah.62.1408489551336; Tue, 19 Aug 2014 16:05:51 -0700 (PDT)
In-Reply-To: <3fb3b4d1-a7e2-4912-a878-7d5e1798aee6@googlegroups.com>
References: <51dfbe9b-f6e0-4532-bc2d-e7ce2fc282b5@googlegroups.com> <mailman.13116.1408398154.18130.python-list@python.org> <3fb3b4d1-a7e2-4912-a878-7d5e1798aee6@googlegroups.com>
Date: Tue, 19 Aug 2014 16:05:51 -0700
Subject: Re: efficient partial sort in Python ?
From: Dan Stromberg <drsalists@gmail.com>
To: Chiu Hsiang Hsu <wdv4758h@gmail.com>
Content-Type: text/plain; charset=UTF-8
Cc: Python List <python-list@python.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.13175.1408489553.18130.python-list@python.org>
Lines: 39
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1408489553 news.xs4all.nl 2864 [2001:888:2000:d::a6]:50006
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:76618

On Tue, Aug 19, 2014 at 12:37 PM, Chiu Hsiang Hsu <wdv4758h@gmail.com> wrote:
> On Tuesday, August 19, 2014 5:42:27 AM UTC+8, Dan Stromberg wrote:
>> On Mon, Aug 18, 2014 at 10:18 AM, Chiu Hsiang Hsu <wdv4758h@gmail.com> wrote:
>>
>> > I know that Python use Timsort as default sorting algorithm and it is efficient,
>>
>> > but I just wanna have a partial sorting (n-largest/smallest elements).
>>
>>
>>
>> Perhaps heapq with Pypy?  Or with nuitka?  Or with numba?

> Another problem with heapq is the memory usage, it cost a lot of more memory with heapq in CPython (I test it in 3.4 with 1000000 float numbers) compare to sorted.

This surprises me.  I believe heapq probably keeps values in a python
list with no extra references, by making node i's left child and right
child be array elements 2*i and 2*i+1, respectively.

A heap of some sort probably is best algorithmically.  You're probably
just up against a high constant.  On the other hand, there are many
kinds of heaps.

> For curiosity, there are many speed up solution in Python (like Cython, PyPy), I hasn't use Cython before,
> I guess PyPy is a more convient way to speed up current Python code (?),
> so how does Cython compare to PyPy ? (speed, code, flexibility, or anything else)

PyPy is really fast for CPU-intensive workloads, but CPython is better for I/O.

I tested a single CPU-intensive microbenchmark of Cython and PyPy
(also Jython and CPython).  PyPy was fastest
(http://stromberg.dnsalias.org/~strombrg/backshift/documentation/performance/index.html).

I haven't yet compared numba or nuitka or Shedskin.

When you use heapq, are you putting all the values in the heap, or
just up to n at a time (evicting the worst value, one at a time as you
go)?  If you're doing the former, it's basically a heapsort which
probably won't beat timsort.  If you're doing the latter, that should
be pretty good.