Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #43546

Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ?

Show all headers | View raw

On 04/13/2013 10:01 PM, Dennis Lee Bieber wrote:
> On Sun, 14 Apr 2013 00:03:25 +1000, Chris Angelico <rosuav@gmail.com>
> declaimed the following in gmane.comp.python.general:

[ ....]

>> * Create a table with a number of rows with an ID and a counter,
>> initialized to 0
>> * Repeatedly, in parallel, perform a transaction:
>>    - Increment the counter on one of the rows (at random)
>>    - Increment a "possible" in-memory counter for that row
>>    - Commit the database transaction
>>    - Increment a "confirmed" in-memory counter for that row
>> * When an error of "database seems to be down" is detected, wait for
>> it to come up again, then query the table. The counters must all be at
>> least their corresponding "possible" value and at most the
>> "confirmed".
>>
> 	SQLite is a "file server" database (like M$ JET engine [aka:
> "Access"]). It's locking system is multi-stage. It allows multiple
> concurrent readers on a "shared" lock state. Only one connection can
> perform write operations ("reserved" lock) alongside the readers. A
> second connection attempting to perform a write will be rejected with a
> database locked condition. Then it really gets nasty -- the writer
> attempts to commit the update: The first step is to block other
> connections from even entering the read state (the "pending" lock).
> However, the writer itself is blocked until all remaining readers have
> exited; only then does it have exclusive access to and SQLite makes
> changes to the database file itself (prior to that, the writer
> connection is changing page images in memory)

Ok, this makes sense... It's not something I'll bother about to begin 
with, but maybe later (for critical apps) I can see that this is important.

[ ....]

> 	In the commit phase, SQLite first tries to ensure the rollback
> journal is flushed to disk -- but that apparently is out of its control;
> it can submit a sync command to the OS, but has to rely on what the OS
> tells it about the state of the writes to disk (the book indicates that
> some IDE drives would lie when queried about sync status, while still
> having unwritten data in the on-board buffers). After the rollback
> journal it submits the data to the database. I

I agree, this must be a problem, when the OS is lying...

> Crash during journal write: restart finds no journal, that transaction
> is lost but the database itself is clean
>
> Crash after journal during database update, restart finds journal,
> assumes database is suspect, and rolls back the pages, database is
> restored to pre-transaction state
>
> Crash after database sync during removal of journal, restart either
> finds journal still there and rolls back the pages restoring to
> pretransaction state, or the file was removed from the directory and
> SQLite determines database file is good with the last transaction in
> place.

Ok, this is a bit more advanced - I'll try to make my own experiments 
now and then after some time I guess I can dig more into these details, 
thanks.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-11 00:06 +0200
  Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Cousin Stanley <cousinstanley@gmail.com> - 2013-04-11 01:39 +0000
    Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-11 10:49 +0200
      Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-11 15:38 +0200
  Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Cousin Stanley <cousinstanley@gmail.com> - 2013-04-11 17:58 +0000
    Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Cousin Stanley <cousinstanley@gmail.com> - 2013-04-11 18:44 +0000
      Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-12 16:19 +0200
    Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-11 21:42 +0200
    Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-12 16:03 +0200
      Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Cousin Stanley <cousinstanley@gmail.com> - 2013-04-12 16:58 +0000
        Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-12 22:52 +0200
          Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Cousin Stanley <cousinstanley@gmail.com> - 2013-04-12 23:26 +0000
            Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 02:00 +0200
            Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-13 01:44 +0000
              Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 13:08 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-13 21:39 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 15:30 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-14 00:03 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Roy Smith <roy@panix.com> - 2013-04-13 10:36 -0400
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 21:25 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Roy Smith <roy@panix.com> - 2013-04-13 17:38 -0400
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 16:39 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Walter Hurry <walterhurry@lavabit.com> - 2013-04-13 14:56 +0000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 21:34 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Walter Hurry <walterhurry@lavabit.com> - 2013-04-13 22:22 +0000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-14 00:31 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-14 08:54 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-14 02:06 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-14 08:34 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-14 02:10 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-14 02:15 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? rusi <rustompmody@gmail.com> - 2013-04-13 10:02 -0700
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 21:49 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-14 07:56 +0000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? rusi <rustompmody@gmail.com> - 2013-04-14 04:17 -0700
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-14 23:22 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? rusi <rustompmody@gmail.com> - 2013-04-15 04:45 -0700
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-15 22:28 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Ned Deily <nad@acm.org> - 2013-04-14 09:40 -0700
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Tim Chase <python.list@tim.thechases.com> - 2013-04-14 15:16 -0500
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Roy Smith <roy@panix.com> - 2013-04-14 17:48 -0400
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-15 07:43 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 21:42 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-04-13 16:01 -0400
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? someone <newsboost@gmail.com> - 2013-04-13 23:36 +0200
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Chris Angelico <rosuav@gmail.com> - 2013-04-14 08:44 +1000
                Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-04-13 19:42 -0400
              Re: python-noob - which container is appropriate for later exporting into mySql + matplotlib ? Cousin Stanley <cousinstanley@gmail.com> - 2013-04-14 18:20 +0000

csiph-web