Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #42317

Re: How to find bad row with db api executemany()?

Show key headers only | View raw

On Sat, Mar 30, 2013 at 3:10 PM, Roy Smith <roy@panix.com> wrote:
> In article <mailman.3986.1364615879.2939.python-list@python.org>,
>  Chris Angelico <rosuav@gmail.com> wrote:
>
>> Side point: You mentioned SSDs. Are you aware of the fundamental risks
>> associated with them? Only a handful of SSD models are actually
>> trustworthy for databasing.
>
> We haven't decided if we're going that route yet, but if we do, we will
> probably use do RAID SSD for added reliability.  We also have all our
> database servers in failover clusters, so we get added reliability that
> way too.

But will you know if you have corruption? Normally, transactional
integrity means:

1) If a transaction, from begin to commit, is not completely applied,
then it is completely not-applied; and
2) If a transaction that was affected by this one has been applied,
then so has this one.

SSDs that lie about fsync (and some hard disks lie too, as do some
operating systems and some file system drivers, but - under Linux at
least - it's possible to guarantee the OS and FS parts) can violate
both halves. Anything might have been written, anything might have
been missed. I did some tests with PostgreSQL on an SSD, and the
results were seriously scary.

> But, we have some runway left with more conventional technologies, so we
> don't need to decide for a while.  Ultimately, however, as reliability
> goes up and cost comes down, it's hard to imagine the SSD isn't going to
> be a big part of our lives at some point.

Yes. I hope that by then, the manufacturers will realize that TPS
isn't the only thing that matters. I'm sure SSDs will mature to the
point where we can trust all brands equally (or at least, most brands
- maybe it'll be like "server SSDs" and "desktop SSDs"?), but until
then, there aren't many options.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-29 18:25 -0400
  Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 20:41 -0400
    Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 11:57 +1100
      Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 21:19 -0400
        Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 13:05 +1100
        Re: How to find bad row with db api executemany()? Tim Chase <python.list@tim.thechases.com> - 2013-03-29 22:17 -0500
        Re: How to find bad row with db api executemany()? (PS) Tim Chase <python.list@tim.thechases.com> - 2013-03-29 22:38 -0500
        Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-29 23:38 -0400
    Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 22:44 -0400
      Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 13:49 +1100
        Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 23:09 -0400
          Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 14:14 +1100
            Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 23:36 -0400
              Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 14:57 +1100
                Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-30 00:10 -0400
                Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 15:21 +1100
                Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-30 10:19 -0400
        Re: How to find bad row with db api executemany()? rusi <rustompmody@gmail.com> - 2013-03-29 20:13 -0700
          Re: How to find bad row with db api executemany()? rusi <rustompmody@gmail.com> - 2013-03-29 20:15 -0700
          Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 23:40 -0400
      Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-29 23:53 -0400
      Re: How to find bad row with db api executemany()? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-03-30 00:19 -0400
      Re: How to find bad row with db api executemany()? Chris Angelico <rosuav@gmail.com> - 2013-03-30 15:24 +1100
      Re: How to find bad row with db api executemany()? Tim Chase <python.list@tim.thechases.com> - 2013-03-30 06:38 -0500

csiph-web