Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #42247 > unrolled thread

How to find bad row with db api executemany()?

Back to article view | Back to comp.lang.python

Contents

  How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-29 10:48 -0400
    Re: How to find bad row with db api executemany()? Miki Tebeka <miki.tebeka@gmail.com> - 2013-03-30 09:24 -0700
      Re: How to find bad row with db api executemany()? Roy Smith <roy@panix.com> - 2013-03-30 12:44 -0400

#42247 — How to find bad row with db api executemany()?

I'm inserting a gazillion rows into a MySQL database using MySQLdb and cursor.executemany() for efficiency.  Every once in a while, I get a row which violates some kind of database constraint and raises Error.

I can catch the exception, but don't see any way to tell which row caused the problem.  Is this information obtainable, short of retrying each row one by one?

---
Roy Smith
roy@panix.com

[toc] | [next] | [standalone]

#42338

> I can catch the exception, but don't see any way to tell which row caused the problem.  Is this information obtainable, short of retrying each row one by one?
One way to debug this is to wrap the iterable passed to executemany with one that remembers the last line. Something like:

    class LastIterator(object):
        def __init__(self, coll):
            self.it = iter(coll)
            self.last = None

        def __iter__(self):
            return self

        def next(self):
            self.last = next(self.it)
            return self.last

      ...
      li = ListIterator(items)
      try:
           cursor.executemany(sql, li)
      except SQLError, e:
           print('Error: {}, row was {}'.format(e, li.last))

[toc] | [prev] | [next] | [standalone]

#42342

In article <6b085f98-0d19-4572-be92-4fb1764a221b@googlegroups.com>,
 Miki Tebeka <miki.tebeka@gmail.com> wrote:

> > I can catch the exception, but don't see any way to tell which row caused 
> > the problem.  Is this information obtainable, short of retrying each row 
> > one by one?
> One way to debug this is to wrap the iterable passed to executemany with one 
> that remembers the last line. Something like:
> 
>     class LastIterator(object):
>         def __init__(self, coll):
>             self.it = iter(coll)
>             self.last = None
> 
>         def __iter__(self):
>             return self
> 
>         def next(self):
>             self.last = next(self.it)
>             return self.last
> 
>       ...
>       li = ListIterator(items)
>       try:
>            cursor.executemany(sql, li)
>       except SQLError, e:
>            print('Error: {}, row was {}'.format(e, li.last))

This assumes that the exception is raised synchronously with iterating 
over the input.  The whole idea of executemany() is to batch up rows and 
send them to the database as a single unit, so this would almost 
certainly not be a good assumption.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web