Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.advocacy > #471306
| From | owl <owl@rooftop.invalid> |
|---|---|
| Newsgroups | comp.os.linux.advocacy |
| Subject | Re: slrn is trembling |
| Date | 2018-10-09 17:47 +0000 |
| Organization | O.W.L. |
| Message-ID | <yuta77a.e29ga@rooftop.invalid> (permalink) |
| References | (6 earlier) <bbd700fd-6dff-4700-adf4-aa725ba29a6b@googlegroups.com> <pk9a99ba3.l4a@rooftop.invalid> <a3ce18b3-1ab8-4a63-bfd9-c8c01a3ed760@googlegroups.com> <8z98bbkea98.l97@rooftop.invalid> <ddb7e1e2-7406-4872-b7b0-0365b2599e1d@googlegroups.com> |
Steve Carroll <fretwizzer@gmail.com> wrote:
> On Tuesday, October 9, 2018 at 9:55:11 AM UTC-6, owl wrote:
>> Steve Carroll <fretwizzer@gmail.com> wrote:
>> > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote:
>> >> Steve Carroll <fretwizzer@gmail.com> wrote:
>> >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote:
>> >> >> Steve Carroll <fretwizzer@gmail.com> wrote:
>> >> >
>>
>> >>
>> >>
>> >> It doesn't work right for a large number of posts, and there's a problem
>> >> with the csplit regex, so I'm scrapping the whole shell approach. Too
>> >> much labor to make it not messy, and I don't trust bash arrays for
>> >> large amounts of big data.
>> >
>> > I was thinking of refreshing the array with the processing and writing
>> > of each record but I haven't yet figured out how while keeping things
>> > in sync.
>>
>>
>> I got all the data in an array of structs that have integer member
>> for the article id.
>
> Even the bodies? How'd you determine how to size the structs for those
> (don't structs need to be sized up front)?
>
I'm just massively over-allocating right now. I've been focusing on
other parts of the code. I'll go back later and size everything closer
to what's necessary.
struct post
{
int articleid;
char *fromheader;
char *article;
char *headers;
char *body;
};
struct post *parray;
I'll expand the struct later for more stuff later.
>> > What's the problem with large bash arrays?
>> >
>>
>> They're just messy. And I don't know if they are go on the stack
>> or not.
>>
>>
>> >> I'm working on one in C, with everything in RAM.
>> >
>> > To me that makes more sense (why pound a drive when you have RAM
>> > which is faster anyway?). I was messing with C before I started with this
>> > stuff again but, from what little I saw, the syntax wasn't as grueling as
>> > Bash. I was on step one with C... I didn't get to the point where I was
>> > using system() or anything like that so I'd have to figure out how to get
>> > all these little utility apps (awk, sed, cut, etc.) working in it if I were to
>> > go this route. If you are giving up on Bash for this that tells me a lot.
>> > In any event, I don't regret having spent time with it as it's always
>> > been on a list of things to mess with. I read your post where you say
>> > you get drawn off to the next thing... I know that feeling well.
>> >
>>
>> Not using system(). The code does all parsing and writes.
>> Right now it's just walking the bytes and comparing, but
>> I'm considering using the regex library. Header extraction
>> is easy enough without it though.
>
> Are you comparing to the header labels and the newline char? What
> about References: and the body content? Many of the posts have a
> line that begins: On... (some date, someone) wrote:
>
I'm basically just walking to a newline in the headers, then looking
char-by-char for a header match after the newline. At that point, I've
already split the posts into headers and body in a struct (based on
'^.$' delimiter), so there's only going to be one header match for each
struct's headers member. For example, parray[0].headers will contain
only one "From:" header match. The code is a tad wordy this way,
but it works. Like I said, I may go back and do it with the regex
library. I need some practice with that anyway.
> But there are posts that don't have it or anything you could use. I was
> thinking about blanks lines after the final header, which all seem to use
> angle brackets (but I could be wrong about that).
>
It doesn't matter. It reads the file into a struct's article member
until it hits a newline, then checks if the next char is a '.', and
if so, then is the next char another newline. If so, it increments
the article count and goes to the next struct. Once all the structs
have their article members filled, it goes back and parses the articles
into headers and body (breaking on the first blank line) and fills the
headers and body members. After that, it's just a matter of walking each
struct's headers member and testing for whatever header you want to match.
That's my approach, anyway. There may be a better way.
>> >> With a 1000-post file,
>> >> it does inserts to the fromtable and headers table, and takes about
>> >> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me.
>> >> The inserts are a real performance bottleneck. How do JS and Python
>> >> do for insert time?
>> >
>> > I dunno, I've never done this much data so I've had no need to check.
>> > DFS is the guy to ask for Python. I've heard the current version of Php
>> > is relatively fast at this, Sandman could weigh in on specifics for it.
>> >
>> > It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims:
>> > "To my surprise, LOAD DATA INFILE proves faster than a table copy:"
>> >
>> > <https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723>
>>
>> Holy shit! I wrapped the inserts in a transaction and now it does 2000
>> inserts in half a second.
>
> From 22/sec to 1000/sec increase, not too shabby.
>
>> And that's big inserts -- full headers in each
>> record for 1000 of them.
>
> To multiple tables?
>
Yes. I added full body inserts, giving it another 1000 or so big inserts,
and it pushed the time up to 0.5 seconds. :( :)
>> I think DFS gave me that pointer once before,
>> and I forgot it.
>
> Getting them as fast as you can via a shell script and writing them
> like this is definitely a solid way to go for speed. It's sorta looking
> like you could get a year's worth of posts, process 'em and write it
> all to a DB in under a minute.
>
Could be.
Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 10:08 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 08:51 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 13:50 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 11:59 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 21:23 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:12 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:21 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:55 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 13:51 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 11:16 -0700
"Path:" is the only line that's never fake. Jeff-Relf.Me @. - 2018-10-14 01:41 -0700
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 11:16 -0400
Re: "Path:" is the only line that's never fake. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 08:38 -0700
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 16:40 +0100
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 11:43 -0400
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 17:12 +0100
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 12:23 -0400
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Wader of Doom) - 2018-10-14 18:01 +0100
No one can do "a perfect forgery" of my posts. Jeff-Relf.Me @. - 2018-10-14 14:29 -0700
Re: No one can do "a perfect forgery" of my posts. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 17:33 -0400
Re: No one can do "a perfect forgery" of my posts. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 22:47 +0100
Re: No one can do "a perfect forgery" of my posts. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 17:53 -0400
Re: No one can do "a perfect forgery" of my posts. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 22:58 +0100
Re: No one can do "a perfect forgery" of my posts. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 14:56 -0700
Re: No one can do "a perfect forgery" of my posts. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 14:41 -0700
Re: "Path:" is the only line that's never fake. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 08:54 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-14 11:26 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-14 09:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-14 18:53 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-14 13:27 -0700
Re: slrn is trembling Melzzzzz <Melzzzzz@zzzzz.com> - 2018-10-14 20:55 +0000
Pipelining "Article" commands. Jeff-Relf.Me @. - 2018-10-14 14:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-15 13:49 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 07:34 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-15 21:54 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 19:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 21:51 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 03:16 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 20:33 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-16 10:26 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 07:44 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 15:17 +0000
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-16 12:26 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-16 09:46 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 10:21 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 14:00 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 16:49 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 12:35 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 20:44 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 14:09 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 14:50 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 22:32 +0000
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 23:03 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 16:28 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 16:54 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:02 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:20 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:31 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:32 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:54 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:02 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:07 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 18:11 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:48 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 02:06 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:01 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 15:35 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:37 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:53 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:57 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:17 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:19 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 09:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 16:19 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 16:40 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 17:39 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 11:07 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 18:14 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 11:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 18:52 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 08:46 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:29 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:48 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:51 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-06 12:19 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 15:34 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-08 21:27 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-08 17:49 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-08 18:13 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 07:31 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 13:45 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 07:55 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 15:55 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 09:20 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 09:39 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 09:41 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 10:23 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 23:38 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 17:47 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 11:09 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-10 09:04 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 13:56 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-11 05:02 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:33 -0400
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-10 19:21 +0000
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-12 20:46 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 23:50 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-08 18:44 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-06 14:08 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 11:32 -0700
Re: slrn is trembling "F. Russell" <fr@random.info> - 2018-10-05 00:32 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:46 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:55 +0000
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:17 +0000
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-05 06:59 -0500
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 05:14 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 12:48 +0000
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-05 11:17 -0500
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 13:35 -0400
Re: slrn is trembling Silicon Vee <siliconvee1945@yahoo.com> - 2018-10-11 01:36 +0200
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:07 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:12 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:21 -0700
Re: slrn is trembling SilverSlimer <silver@slim.er> - 2018-10-05 12:01 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 05:30 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:13 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 12:50 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 10:35 -0700
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-10 08:51 -0500
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 07:09 -0700
Re: slrn is trembling Sam O'Nella <poison@donteatit.net> - 2018-10-05 16:43 -0400
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:34 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 07:56 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 12:38 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-11 02:12 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-12 12:27 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 12:42 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 21:47 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 03:28 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 20:45 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-05 15:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 15:41 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 19:21 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-05 13:44 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 17:48 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 16:13 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 20:47 -0400
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:09 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:18 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 21:38 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 19:04 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 02:05 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 19:35 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:36 -0400
csiph-web