Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #27988

Re: Python 2.6 and Sqlite3 - Slow

Show all headers | View raw

Am 27.08.2012 03:23, schrieb bruceg113355@gmail.com:
> My program uses Python 2.6 and Sqlite3 and connects to a network
> database 100 miles away.

Wait, isn't SQLite completely file-based? In that case, SQLite accesses
a file, which in turn is stored on a remote filesystem. This means that 
there are other components involved here, namely your OS, the network 
(bandwidth & latency), the network filesystem and the filesystem on the 
remote machine. It would help if you told us what you have there.


> My program reads approx 60 records (4000 bytes) from a Sqlite
> database in less than a second. Each time the user requests data, my
> program can continuously read 60 records in less than a second.
> However, if I access the network drive  (e.g. DOS command DIR /S)
> while my program is running, my program takes 20 seconds to read the
> same 60 records. If I restart my program, my program once again takes
> less than a second to read 60 records.

Questions here:
1. Is each record 4kB or are all 60 records together 4kB?
2. Does the time for reading double when you double the number of 
records? Typically you have B + C * N, but it would be interesting to 
know the bias B and the actual time (and size) of each record.
3. How does the timing change when running dir/s?
4. What if you run two instances of your program?
5. Is the duration is only reset by restarting the program or does it 
also decrease when the dir/s call has finished? What if you close and 
reopen the database without terminating the program?

My guess is that the concurrent access by another program causes the 
accesses to become synchronized, while before most of the data is 
cached. That would cause a complete roundtrip between the two machines 
for every access, which can easily blow up the timing via the latency.

In any case, I would try Python 2.7 in case this is a bug that was 
already fixed.

Good luck!

Uli

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-26 18:23 -0700
  Re: Python 2.6 and Sqlite3 - Slow Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-08-27 17:50 +0200
    Re: Python 2.6 and Sqlite3 - Slow ahsanraza211@gmail.com - 2012-08-27 09:53 -0700
    Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-27 13:41 -0700
      Re: Python 2.6 and Sqlite3 - Slow Cameron Simpson <cs@zip.com.au> - 2012-08-28 18:27 +1000
        Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-28 10:25 -0700
          Re: Python 2.6 and Sqlite3 - Slow Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-28 17:20 -0400
          Re: Python 2.6 and Sqlite3 - Slow Pedro Larroy <pedro.larroy.lists@gmail.com> - 2012-08-29 00:01 +0200
        Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-28 10:25 -0700
  Re: Python 2.6 and Sqlite3 - Slow Demian Brecht <demianbrecht@gmail.com> - 2012-08-27 13:54 -0700
    Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-27 17:55 -0700
      Re: Python 2.6 and Sqlite3 - Slow Bryan <bryanjugglercryptographer@yahoo.com> - 2012-08-27 19:32 -0700
        Re: Python 2.6 and Sqlite3 - Slow bruceg113355@gmail.com - 2012-08-27 20:09 -0700
          Re: Python 2.6 and Sqlite3 - Slow Bryan <bryanjugglercryptographer@yahoo.com> - 2012-08-27 22:26 -0700
          Re: Python 2.6 and Sqlite3 - Slow Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-28 01:33 -0400

csiph-web