Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.advocacy > #413279

Re: Snit hitting the glue bag early today

From owl <owl@rooftop.invalid>
Newsgroups comp.os.linux.advocacy
Subject Re: Snit hitting the glue bag early today
Date 2017-05-06 04:48 +0000
Organization O.W.L.
Message-ID <pa80b.3yvevv45gv7j@rooftop.invalid> (permalink)
References (11 earlier) <oegs1s$b3k$3@dont-email.me> <acv993.by93ap@rooftop.invalid> <oeias1$v75$2@dont-email.me> <zvz8v03.bu8ie@rooftop.invalid> <oejgv5$btg$2@dont-email.me>

Show all headers | View raw

DFS <nospam@dfs.com> wrote:
> On 5/5/2017 4:14 PM, owl wrote:
> 
>> C creating a csv and using "copy from" instead of inserts:
>> anon@lowtide:~/code/pg$ time ./flah >/dev/null
>> PGRES_COMMAND_OK
>> 
>> real    0m0.111s
>> user    0m0.008s
>> sys     0m0.000s
>> anon@lowtide:~/code/pg$
>> 
>> Without using "copy from", the Python and C have similar performance.
>> The postgresql connection is the bottleneck.  However, depending on the
>> complexity and volume of work necessary beforehand to build the queries,
>> C could easily stomp the crap out of Python.  :)
> 
> 
> Write .csv then COPY FROM it is sneaky.  You must've gotten desperate 
> when it was 8 seconds to insert 1K rows.
> 

heh

> But I didn't specify INSERTs so it's legal for the challenge.
> 
> After doing the COPY FROM, are you writing back out the summary data to 
> a new .csv (last requirement)?
> 

Yes.  Actually doing two queries and writing to a names.csv and
dates.csv.

anon@lowtide:~/code/pg$ cat names.csv
"trixie","221"
"norton","186"
"ralph","210"
"alice","179"
"moe","204"
anon@lowtide:~/code/pg$ head dates.csv
"2017-01-01","29"
"2017-01-13","29"
"2017-01-26","38"
"2017-01-04","41"
"2017-01-03","24"
"2017-01-30","30"
"2017-01-27","36"
"2017-01-19","32"
"2017-01-10","45"
"2017-01-15","32"
anon@lowtide:~/code/pg$ 

> my previous python: 0.28s  (0.226s on your system)
> new python (writes random data to file, imports it via StringIO module)
> 
> $ python colatest3.py 1000
> 2017-05-25 45
> 2017-05-03 44
> 2017-05-15 44
> 2017-05-21 40
> 2017-05-18 39
> 2017-05-01 38
> 2017-05-27 38
> 2017-05-30 38
> 2017-05-26 36
> 2017-05-13 35
> ...
> .csv file written
> 
> 0.127s prog exec
> 
> 
> 50% faster. hoo-ahh.
> 
> Please test it on your system and see what kind of beatdown it puts on 
> your C  (note: to save bytes I'm only outputting 10 rows of summary 
> data, as did the previous code)
> 

When I run it, it doesn't write to the db and there is no
record of the csv file that contained the 1000 records,
which appears to be the same file you write your summary
to, so... 

The first code you had worked for writing to the db.
Also, the timer function you are using shows about 1/10
the usage that "time" reports as "real".

anon@lowtide:~/code/pg$ time ./dfs_copy 1000
2017-05-27 44
2017-05-04 42
2017-05-21 41
2017-05-26 40
2017-05-01 39
2017-05-30 39
2017-05-12 37
2017-05-15 36
2017-05-20 35
2017-05-05 34
...
.csv file written

0.020s prog exec

real    0m0.231s
user    0m0.096s
sys     0m0.008s
anon@lowtide:~/code/pg$ 

dummydb=# \dt
No relations found.
dummydb=#

The only changes I made were to dbname and the path for colatest.csv.

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-04 09:29 -0400
  Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-04 16:42 +0000
    Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-04 20:06 +0000
      Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-04 20:12 +0000
        Re: Snit hitting the glue bag early today chrisv <chrisv@nospam.invalid> - 2017-05-05 06:52 -0500
          Re: Snit hitting the glue bag early today Marek Novotny <marek.novotny@marspolar.com> - 2017-05-05 10:32 -0500
            Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:22 -0400
          Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:01 -0400
      Re: Snit hitting the glue bag early today Steve Carroll <fretwizzer@gmail.com> - 2017-05-04 13:16 -0700
      Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-04 21:36 +0000
    Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-06 09:44 -0400
  Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-04 19:49 +0000
    Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-04 19:28 -0400
      Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 02:56 +0000
        Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-04 23:40 -0400
          Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 06:18 +0000
            Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 06:34 +0000
              Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-05 07:07 +0000
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 07:29 +0000
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 16:48 +0000
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 20:49 +0000
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:02 -0400
                Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-05 19:12 +0000
                Re: Snit hitting the glue bag early today Snit <usenet@gallopinginsanity.com> - 2017-05-05 13:45 -0700
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 17:09 -0400
                Re: Snit hitting the glue bag early today Sandman <mr@sandman.net> - 2017-05-05 21:41 +0000
            Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:00 -0400
              Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-05 20:14 +0000
                Re: Snit hitting the glue bag early today Snit <usenet@gallopinginsanity.com> - 2017-05-05 13:44 -0700
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 23:51 -0400
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-06 04:48 +0000
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-06 10:06 -0400
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-06 15:50 +0000
                Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-07 13:09 -0400
                Re: Snit hitting the glue bag early today owl <owl@rooftop.invalid> - 2017-05-07 18:39 +0000
        Re: Snit hitting the glue bag early today Chris Ahlstrom <OFeem1987@teleworm.us> - 2017-05-05 05:28 -0400
          Re: Snit hitting the glue bag early today DFS <nospam@dfs.com> - 2017-05-05 13:02 -0400

csiph-web