Re: How to find bad row with db api executemany()?

Path	csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.056
X-Spam-Evidence	'H': 0.89; 'S': 0.00; 'insert': 0.05; 'model,': 0.05; 'provisioning': 0.09; 'subject:How': 0.10; '"we': 0.16; '(it': 0.16; 'bus.': 0.16; 'dump': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'guys,': 0.16; 'hmm.': 0.16; 'roy': 0.16; 'statements,': 0.16; 'why,': 0.16; 'sat,': 0.16; 'wrote:': 0.18; 'fit': 0.20; 'cloud': 0.22; 'documented': 0.24; 'replace': 0.24; 'initial': 0.24; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'waste': 0.30; 'especially': 0.30; 'message- id:@mail.gmail.com': 0.30; 'went': 0.31; 'usually': 0.31; 'loads': 0.31; "we're": 0.32; 'amazon': 0.34; 'could': 0.34; 'subject:with': 0.35; 'basic': 0.35; 'received:209.85': 0.35; 'received:209.85.220': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'disk': 0.36; 'explains': 0.36; 'doing': 0.36; "didn't": 0.36; 'received:209': 0.37; 'performance': 0.37; 'to:addr:python-list': 0.38; 'pm,': 0.38; 'that,': 0.38; 'heard': 0.39; 'to:addr:python.org': 0.39; 'even': 0.60; 'money.': 0.60; 'decided': 0.64; 'our': 0.64; 'more': 0.64; 'great': 0.65; '2000': 0.65; '30,': 0.65; 'mar': 0.68; 'forums': 0.68; 'shortly': 0.68; 'smith': 0.68; 'article': 0.77; '"hey,': 0.84; 'insert,': 0.84; 'subject:find': 0.84; 'bandwidth': 0.96; '2013': 0.98
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=o9l3wLT3MVk9SEvoLXNQgDlHY2QvcQVpOKrfCniOf7g=; b=JzBsVqBokRA1NvSnvbY39EXCV+6+6olFHkikF2t8BXjJ2WcE0YVXXDJY5JNVwO4OAj 400HHNlAGgix1ABpnxT/uzcgdL0k9U1nyNuczy8JV9Y7HidIzHcWhn0ly6vZ6LOubU0o mb/wtAd3ulsE4lIMIIXaJbLVZd+BmPEauFngDYtzJOB97N62LGnazdcky5+9Pe+3BnJ5 0PjQ7gVm82+wAB4+WbR76IFYl8leN+LKlIZnN6g6MocUSNNaUc6F8aAyI6hJQn2x1hNI iKjV1842z2Qpqdvfo0sUOVSJSQDITz7OnqEtK3aq97cwvDpNKgJ9+A2+FGGgxOckFVhx 6huQ==
MIME-Version	1.0
X-Received	by 10.52.67.164 with SMTP id o4mr2942559vdt.42.1364613278253; Fri, 29 Mar 2013 20:14:38 -0700 (PDT)
In-Reply-To	<roy-8D4A5A.23093729032013@news.panix.com>
References	<F40EFF8D-9F4F-4ACB-8671-450F3CD761CA@panix.com> <5155E32A.1000403@davea.name> <mailman.3971.1364595940.2939.python-list@python.org> <roy-A61512.20410329032013@news.panix.com> <roy-1FF146.22445329032013@news.panix.com> <mailman.3980.1364611804.2939.python-list@python.org> <roy-8D4A5A.23093729032013@news.panix.com>
Date	Sat, 30 Mar 2013 14:14:38 +1100
Subject	Re: How to find bad row with db api executemany()?
From	Chris Angelico <rosuav@gmail.com>
To	python-list@python.org
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.3981.1364613280.2939.python-list@python.org> (permalink)
Lines	27
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1364613280 news.xs4all.nl 6886 [2001:888:2000:d::a6]:49770
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:42306

Show key headers only | View raw

On Sat, Mar 30, 2013 at 2:09 PM, Roy Smith <roy@panix.com> wrote:
> In article <mailman.3980.1364611804.2939.python-list@python.org>,
>  Chris Angelico <rosuav@gmail.com> wrote:
>
>> Especially facepalm because there's some way to do this that's faster
>> than straight INSERT statements, and it's not clearly documented as
>> "hey, guys, if you want to dump loads of data in, use COPY instead"
>> (it might be that, I don't know, but usually COPY isn't directly
>> transliterable with INSERT).
>
> We're actually using REPLACE INTO.  For the initial data load, we could
> just as well do INSERT, but we need the REPLACE functionality as we roll
> in the daily incremental updates.
>
> This also explains why, even after provisioning our RDS instance for
> 2000 IOPS (that's AWS-speak for "we paid extra to get more disk
> bandwidth"), we didn't see any performance improvement!

Hmm. I heard around the forums that Amazon weren't that great at disk
bandwidth anyway, and that provisioning IO was often a waste of money.
But we never did all that much much research on Amazon I/O
performance; shortly after doing some basic benchmarking, we decided
that the cloud was a poor fit for our system model, and went looking
at dedicated servers with their own RAID storage right there on the
bus.

ChrisA

Thread

Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-29 18:25 -0400
  Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 20:41 -0400
    Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 11:57 +1100
      Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 21:19 -0400
        Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 13:05 +1100
        Re: How to find bad row with db api executemany()? Tim Chase <python.list@tim.thechases.com> - 2013-03-29 22:17 -0500
        Re: How to find bad row with db api executemany()? (PS) Tim Chase <python.list@tim.thechases.com> - 2013-03-29 22:38 -0500
        Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-29 23:38 -0400
    Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 22:44 -0400
      Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 13:49 +1100
        Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 23:09 -0400
          Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 14:14 +1100
            Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 23:36 -0400
              Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 14:57 +1100
                Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-30 00:10 -0400
                Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 15:21 +1100
                Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-30 10:19 -0400
        Re: How to find bad row with db api executemany()? rusi <rustompmody@gmail.com> - 2013-03-29 20:13 -0700
          Re: How to find bad row with db api executemany()? rusi <rustompmody@gmail.com> - 2013-03-29 20:15 -0700
          Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 23:40 -0400
      Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-29 23:53 -0400
      Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-30 00:19 -0400
      Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 15:24 +1100
      Re: How to find bad row with db api executemany()? Tim Chase <python.list@tim.thechases.com> - 2013-03-30 06:38 -0500

csiph-web