Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!eternal-september.org!feeder.eternal-september.org!border3.nntp.ams.giganews.com!border1.nntp.ams.giganews.com!nntp.giganews.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'subject:bug': 0.05; '(so': 0.07; '21,': 0.07; 'subject: + ': 0.07; 'subject:sqlite3': 0.07; 'received:mail-vc0-f174.google.com': 0.09; 'subject:Python3': 0.09; 'def': 0.10; 'dec': 0.15; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'result:': 0.16; 'true:': 0.16; 'wrote:': 0.17; 'yield': 0.17; 'affected': 0.22; 'header:In-Reply- To:1': 0.25; 'wondering': 0.26; 'am,': 0.27; 'possibly': 0.27; 'separate': 0.27; 'message-id:@mail.gmail.com': 0.27; 'received:209.85.220.174': 0.29; "i'm": 0.29; 'connection': 0.30; 'fri,': 0.30; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'received:209.85.220': 0.35; 'subject:?': 0.35; 'received:209.85': 0.35; 'but': 0.36; 'two': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'performance': 0.39; 'to:addr:python.org': 0.39; 'header:Received:5': 0.40; 'your': 0.60; 'improvements': 0.65; 'subject:Where': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=KGBtecOd12kG3DIM4ehQObwmBCSm3cwzzh9TVBiTmOs=; b=FFYGnrgGd7uQ/lrsbChJMsWAycuDC+a5ZdBpyQuUCiRlIqAkrzbwLYd7Tynjyg2mGl ghdfoqrnlQjECfx4UgJkS0DjfOTyVf1yf7qleCqEchQfp0gQiOfdDcFBKw04wqGnND7R JjX+bcFZOskVf1NVm7bv3lo9UIVV2BqVoq0MwiySr0k6bc+qybqu0wh+0+UzIvDvB0el jdebwunEqJNCToceqFad1XOA0XyixHDtGFxXq/+6v2U/lHc/xs0hFVjPpUNA50rRJQqk X2XjdLu0ckDoY4COjzXq6zwyXXJDvi+cs+CU0OxeO5x9sA0Nd5qa4DA1egGc6yMK/D8s ZFyg==
MIME-Version: 1.0
In-Reply-To: <kav8ni$f38$1@news.albasani.net>
References: <kav8ni$f38$1@news.albasani.net>
Date: Fri, 21 Dec 2012 02:05:31 +1100
Subject: Re: Python3 + sqlite3: Where's the bug?
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.1100.1356015940.29569.python-list@python.org>
Lines: 19
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1356015940 news.xs4all.nl 6971 [2001:888:2000:d::a6]:55753
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:35201

On Fri, Dec 21, 2012 at 1:52 AM, Johannes Bauer <dfnsonfsduifb@gmx.de> wrote:
> def fetchmanychks(cursor):
>         cursor.execute("SELECT id FROM foo;")
>         while True:
>                 result = cursor.fetchmany()
>                 if len(result) == 0:
>                         break
>                 for x in result:
>                         yield x

I'm not familiar with sqlite, but from working with other databases,
I'm wondering if possibly your commits are breaking the fetchmany.

Would it spoil your performance improvements to do all the fetchmany
calls before yielding anything? Alternatively, can you separate the
two by opening a separate database connection for the foo-reading (so
it isn't affected by the commit)?

ChrisA