Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.advocacy > #471296
| From | owl <owl@rooftop.invalid> |
|---|---|
| Newsgroups | comp.os.linux.advocacy |
| Subject | Re: slrn is trembling |
| Date | 2018-10-09 15:55 +0000 |
| Organization | O.W.L. |
| Message-ID | <8z98bbkea98.l97@rooftop.invalid> (permalink) |
| References | (5 earlier) <76a6087a-7aa7-4721-a3b1-8b213987883a@googlegroups.com> <80z08z0b3.agbo3@rooftop.invalid> <bbd700fd-6dff-4700-adf4-aa725ba29a6b@googlegroups.com> <pk9a99ba3.l4a@rooftop.invalid> <a3ce18b3-1ab8-4a63-bfd9-c8c01a3ed760@googlegroups.com> |
Steve Carroll <fretwizzer@gmail.com> wrote: > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote: >> Steve Carroll <fretwizzer@gmail.com> wrote: >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote: >> >> Steve Carroll <fretwizzer@gmail.com> wrote: >> > >> >> >> It doesn't work right for a large number of posts, and there's a problem >> with the csplit regex, so I'm scrapping the whole shell approach. Too >> much labor to make it not messy, and I don't trust bash arrays for >> large amounts of big data. > > I was thinking of refreshing the array with the processing and writing > of each record but I haven't yet figured out how while keeping things > in sync. I got all the data in an array of structs that have integer member for the article id. > What's the problem with large bash arrays? > They're just messy. And I don't know if they are go on the stack or not. >> I'm working on one in C, with everything in RAM. > > To me that makes more sense (why pound a drive when you have RAM > which is faster anyway?). I was messing with C before I started with this > stuff again but, from what little I saw, the syntax wasn't as grueling as > Bash. I was on step one with C... I didn't get to the point where I was > using system() or anything like that so I'd have to figure out how to get > all these little utility apps (awk, sed, cut, etc.) working in it if I were to > go this route. If you are giving up on Bash for this that tells me a lot. > In any event, I don't regret having spent time with it as it's always > been on a list of things to mess with. I read your post where you say > you get drawn off to the next thing... I know that feeling well. > Not using system(). The code does all parsing and writes. Right now it's just walking the bytes and comparing, but I'm considering using the regex library. Header extraction is easy enough without it though. >> With a 1000-post file, >> it does inserts to the fromtable and headers table, and takes about >> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me. >> The inserts are a real performance bottleneck. How do JS and Python >> do for insert time? > > I dunno, I've never done this much data so I've had no need to check. > DFS is the guy to ask for Python. I've heard the current version of Php > is relatively fast at this, Sandman could weigh in on specifics for it. > > It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims: > "To my surprise, LOAD DATA INFILE proves faster than a table copy:" > > <https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723> Holy shit! I wrapped the inserts in a transaction and now it does 2000 inserts in half a second. And that's big inserts -- full headers in each record for 1000 of them. I think DFS gave me that pointer once before, and I forgot it. anon@lowtide:~/usenet/database$ ./dropdb anon@lowtide:~/usenet/database$ ./createdb anon@lowtide:~/usenet/database$ time ./blah real 0m0.482s user 0m0.168s sys 0m0.080s anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb mysql> select count(*) from headers; count(*) 998 mysql> select count(*) from fromtable; count(*) 998 mysql> select * from fromtable limit 5; id content 2364171 Steve Carroll <frelwizzen@gmail.com>\n 2364172 RonB <ronb02NOSPAM@gmail.com>\n 2364173 Steve Carroll <frelwizzen@gmail.com>\n 2364174 Steve Carroll <frelwizzen@gmail.com>\n 2364175 benj <benj.fake@gmail.com>\n mysql> quit anon@lowtide:~/usenet/database$ I need to strip those trailing newlines out somewhere in the code.
Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 10:08 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 08:51 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 13:50 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 11:59 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 21:23 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:12 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:21 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:55 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 13:51 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 11:16 -0700
"Path:" is the only line that's never fake. Jeff-Relf.Me @. - 2018-10-14 01:41 -0700
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 11:16 -0400
Re: "Path:" is the only line that's never fake. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 08:38 -0700
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 16:40 +0100
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 11:43 -0400
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 17:12 +0100
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 12:23 -0400
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Wader of Doom) - 2018-10-14 18:01 +0100
No one can do "a perfect forgery" of my posts. Jeff-Relf.Me @. - 2018-10-14 14:29 -0700
Re: No one can do "a perfect forgery" of my posts. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 17:33 -0400
Re: No one can do "a perfect forgery" of my posts. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 22:47 +0100
Re: No one can do "a perfect forgery" of my posts. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 17:53 -0400
Re: No one can do "a perfect forgery" of my posts. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 22:58 +0100
Re: No one can do "a perfect forgery" of my posts. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 14:56 -0700
Re: No one can do "a perfect forgery" of my posts. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 14:41 -0700
Re: "Path:" is the only line that's never fake. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 08:54 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-14 11:26 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-14 09:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-14 18:53 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-14 13:27 -0700
Re: slrn is trembling Melzzzzz <Melzzzzz@zzzzz.com> - 2018-10-14 20:55 +0000
Pipelining "Article" commands. Jeff-Relf.Me @. - 2018-10-14 14:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-15 13:49 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 07:34 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-15 21:54 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 19:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 21:51 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 03:16 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 20:33 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-16 10:26 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 07:44 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 15:17 +0000
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-16 12:26 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-16 09:46 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 10:21 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 14:00 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 16:49 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 12:35 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 20:44 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 14:09 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 14:50 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 22:32 +0000
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 23:03 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 16:28 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 16:54 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:02 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:20 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:31 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:32 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:54 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:02 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:07 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 18:11 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:48 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 02:06 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:01 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 15:35 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:37 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:53 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:57 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:17 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:19 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 09:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 16:19 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 16:40 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 17:39 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 11:07 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 18:14 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 11:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 18:52 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 08:46 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:29 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:48 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:51 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-06 12:19 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 15:34 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-08 21:27 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-08 17:49 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-08 18:13 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 07:31 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 13:45 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 07:55 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 15:55 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 09:20 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 09:39 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 09:41 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 10:23 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 23:38 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 17:47 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 11:09 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-10 09:04 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 13:56 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-11 05:02 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:33 -0400
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-10 19:21 +0000
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-12 20:46 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 23:50 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-08 18:44 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-06 14:08 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 11:32 -0700
Re: slrn is trembling "F. Russell" <fr@random.info> - 2018-10-05 00:32 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:46 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:55 +0000
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:17 +0000
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-05 06:59 -0500
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 05:14 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 12:48 +0000
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-05 11:17 -0500
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 13:35 -0400
Re: slrn is trembling Silicon Vee <siliconvee1945@yahoo.com> - 2018-10-11 01:36 +0200
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:07 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:12 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:21 -0700
Re: slrn is trembling SilverSlimer <silver@slim.er> - 2018-10-05 12:01 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 05:30 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:13 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 12:50 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 10:35 -0700
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-10 08:51 -0500
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 07:09 -0700
Re: slrn is trembling Sam O'Nella <poison@donteatit.net> - 2018-10-05 16:43 -0400
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:34 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 07:56 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 12:38 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-11 02:12 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-12 12:27 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 12:42 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 21:47 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 03:28 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 20:45 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-05 15:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 15:41 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 19:21 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-05 13:44 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 17:48 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 16:13 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 20:47 -0400
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:09 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:18 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 21:38 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 19:04 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 02:05 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 19:35 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:36 -0400
csiph-web