Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #27990

Re: Python 2.6 and Sqlite3 - Slow

Show all headers | View raw

On Monday, August 27, 2012 8:50:15 AM UTC-7, Ulrich Eckhardt wrote:
> Am 27.08.2012 03:23, schrieb bruceg113355@gmail.com:
> 
> > My program uses Python 2.6 and Sqlite3 and connects to a network
> 
> > database 100 miles away.
> 
> 
> 
> Wait, isn't SQLite completely file-based? In that case, SQLite accesses
> 
> a file, which in turn is stored on a remote filesystem. This means that 
> 
> there are other components involved here, namely your OS, the network 
> 
> (bandwidth & latency), the network filesystem and the filesystem on the 
> 
> remote machine. It would help if you told us what you have there.
> 
> 
> 
> 
> 
> > My program reads approx 60 records (4000 bytes) from a Sqlite
> 
> > database in less than a second. Each time the user requests data, my
> 
> > program can continuously read 60 records in less than a second.
> 
> > However, if I access the network drive  (e.g. DOS command DIR /S)
> 
> > while my program is running, my program takes 20 seconds to read the
> 
> > same 60 records. If I restart my program, my program once again takes
> 
> > less than a second to read 60 records.
> 
> 
> 
> Questions here:
> 
> 1. Is each record 4kB or are all 60 records together 4kB?
> 
> 2. Does the time for reading double when you double the number of 
> 
> records? Typically you have B + C * N, but it would be interesting to 
> 
> know the bias B and the actual time (and size) of each record.
> 
> 3. How does the timing change when running dir/s?
> 
> 4. What if you run two instances of your program?
> 
> 5. Is the duration is only reset by restarting the program or does it 
> 
> also decrease when the dir/s call has finished? What if you close and 
> 
> reopen the database without terminating the program?
> 
> 
> 
> My guess is that the concurrent access by another program causes the 
> 
> accesses to become synchronized, while before most of the data is 
> 
> cached. That would cause a complete roundtrip between the two machines 
> 
> for every access, which can easily blow up the timing via the latency.
> 
> 
> 
> In any case, I would try Python 2.7 in case this is a bug that was 
> 
> already fixed.
> 
> 
> 
> Good luck!
> 
> 
> 
> Uli

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-26 18:23 -0700
  Re: Python 2.6 and Sqlite3 - Slow Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-08-27 17:50 +0200
    Re: Python 2.6 and Sqlite3 - Slow ahsanraza211@gmail.com - 2012-08-27 09:53 -0700
    Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-27 13:41 -0700
      Re: Python 2.6 and Sqlite3 - Slow Cameron Simpson <cs@zip.com.au> - 2012-08-28 18:27 +1000
        Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-28 10:25 -0700
          Re: Python 2.6 and Sqlite3 - Slow Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-28 17:20 -0400
          Re: Python 2.6 and Sqlite3 - Slow Pedro Larroy <pedro.larroy.lists@gmail.com> - 2012-08-29 00:01 +0200
        Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-28 10:25 -0700
  Re: Python 2.6 and Sqlite3 - Slow Demian Brecht <demianbrecht@gmail.com> - 2012-08-27 13:54 -0700
    Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-27 17:55 -0700
      Re: Python 2.6 and Sqlite3 - Slow Bryan <bryanjugglercryptographer@yahoo.com> - 2012-08-27 19:32 -0700
        Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-27 20:09 -0700
          Re: Python 2.6 and Sqlite3 - Slow Bryan <bryanjugglercryptographer@yahoo.com> - 2012-08-27 22:26 -0700
          Re: Python 2.6 and Sqlite3 - Slow Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-28 01:33 -0400

csiph-web