Groups > comp.os.linux.advocacy > #413233

Re: Snit hitting the glue bag early today

From	owl <owl@rooftop.invalid>
Newsgroups	comp.os.linux.advocacy
Subject	Re: Snit hitting the glue bag early today
Date	2017-05-05 20:14 +0000
Organization	O.W.L.
Message-ID	<zvz8v03.bu8ie@rooftop.invalid> (permalink)
References	(10 earlier) <oegd8c$27u$3@dont-email.me> <zcv903ahbue.3u8@rooftop.invalid> <oegs1s$b3k$3@dont-email.me> <acv993.by93ap@rooftop.invalid> <oeias1$v75$2@dont-email.me>

Show all headers | View raw

DFS <nospam@dfs.com> wrote:
> On 5/5/2017 2:18 AM, owl wrote:
>> DFS <nospam@dfs.com> wrote:
> 
...

>                f.write("%s,%s,%s\n" % (dt,cnt,nm))
> print "...\n.csv file written"
> 
> db.close()
> conn.close()
> print "\n%.2fs prog exec"%(time.clock() - startmaster)
> --------------------------------------------------------------------
> 
> That's 35 lines, for 'readability' and timing and feedback.
> 
> I can shorten it to 17 lines and still meet the 7 requirements.
> 

Mine's about 60-70 lines now, but I'm still leaving out a lot
of error-checking code that really should be in there.

...

> 
>> I guess these are the queries you want.  I have to say, it's pretty
>> slow.  Nearly 8.5 seconds for a measly 1000 inserts.
> 
> try
> 
> BEGIN TRANSACTION;
> END TRANSACTION;
> or
> BEGIN TRANSACTION;
> conn.commit()
> 

Thanks.  begin/end tansaction did the trick.  I think I remember
that now from something else I was working on way back.
...

>>      fprintf(stdout,"\"%s\"\n",PQgetvalue(result,i,1));
>>    }
>>    PQclear(result);
>>    PQfinish(conn);
>>    fclose(fp1);
>>    fclose(fp2);
>>    return 0;
>> }
>> -----------------------------------------------
> 
> Thanks.  Not as 'unwieldy' as I expected C/postgres to be.
> 
> 

Also got the "copy from" to work now.  Needed to close the file
before doing the copy.  Here's how things show on my machine now:

Python:
anon@lowtide:~/code/pg$ time ./dfs.py 1000 >/dev/null

real    0m0.226s
user    0m0.032s
sys     0m0.032s
anon@lowtide:~/code/pg$

C with inserts wrapped in transaction:
anon@lowtide:~/code/pg$ time ./glah >/dev/null

real    0m0.197s
user    0m0.016s
sys     0m0.008s
anon@lowtide:~/code/pg$

C creating a csv and using "copy from" instead of inserts:
anon@lowtide:~/code/pg$ time ./flah >/dev/null
PGRES_COMMAND_OK

real    0m0.111s
user    0m0.008s
sys     0m0.000s
anon@lowtide:~/code/pg$ 

Without using "copy from", the Python and C have similar performance.
The postgresql connection is the bottleneck.  However, depending on the
complexity and volume of work necessary beforehand to build the queries,
C could easily stomp the crap out of Python.  :)

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-04 09:29 -0400
  Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-04 16:42 +0000
    Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-04 20:06 +0000
      Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-04 20:12 +0000
        Re: Snit hitting the glue bag early today chrisv <chrisv@nospam.invalid> - 2017-05-05 06:52 -0500
          Re: Snit hitting the glue bag early today Marek Novotny <marek.novotny@marspolar.com> - 2017-05-05 10:32 -0500
            Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:22 -0400
          Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:01 -0400
      Re: Snit hitting the glue bag early today Steve Carroll <fretwizzer@gmail.com> - 2017-05-04 13:16 -0700
      Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-04 21:36 +0000
    Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-06 09:44 -0400
  Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-04 19:49 +0000
    Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-04 19:28 -0400
      Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 02:56 +0000
        Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-04 23:40 -0400
          Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 06:18 +0000
            Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 06:34 +0000
              Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-05 07:07 +0000
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 07:29 +0000
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 16:48 +0000
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 20:49 +0000
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:02 -0400
                Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-05 19:12 +0000
                Re: Snit hitting the glue bag early today Snit <usenet@gallopinginsanity.com> - 2017-05-05 13:45 -0700
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 17:09 -0400
                Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-05 21:41 +0000
            Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:00 -0400
              Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 20:14 +0000
                Re: Snit hitting the glue bag early today Snit <usenet@gallopinginsanity.com> - 2017-05-05 13:44 -0700
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 23:51 -0400
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-06 04:48 +0000
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-06 10:06 -0400
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-06 15:50 +0000
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-07 13:09 -0400
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-07 18:39 +0000
        Re: Snit hitting the glue bag early today Chris Ahlstrom <OFeem1987@teleworm.us> - 2017-05-05 05:28 -0400
          Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:02 -0400

csiph-web