Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.advocacy > #470591 > unrolled thread
| Started by | DFS <nospam@nospam.com> |
|---|---|
| First post | 2018-10-04 10:08 -0400 |
| Last post | 2018-10-10 10:36 -0400 |
| Articles | 20 on this page of 158 — 15 participants |
Back to article view | Back to comp.os.linux.advocacy
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 10:08 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 08:51 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 13:50 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 11:59 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 21:23 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:12 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:21 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:55 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 13:51 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 11:16 -0700
"Path:" is the only line that's never fake. Jeff-Relf.Me @. - 2018-10-14 01:41 -0700
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 11:16 -0400
Re: "Path:" is the only line that's never fake. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 08:38 -0700
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 16:40 +0100
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 11:43 -0400
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 17:12 +0100
Re: "Path:" is the only line that's never fake. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 12:23 -0400
Re: "Path:" is the only line that's never fake. snipeco.2@gmail.com (Wader of Doom) - 2018-10-14 18:01 +0100
No one can do "a perfect forgery" of my posts. Jeff-Relf.Me @. - 2018-10-14 14:29 -0700
Re: No one can do "a perfect forgery" of my posts. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 17:33 -0400
Re: No one can do "a perfect forgery" of my posts. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 22:47 +0100
Re: No one can do "a perfect forgery" of my posts. "Sir Gregory Hall, Esq." <greghall@yacht_maester.fake> - 2018-10-14 17:53 -0400
Re: No one can do "a perfect forgery" of my posts. snipeco.2@gmail.com (Sn!pe) - 2018-10-14 22:58 +0100
Re: No one can do "a perfect forgery" of my posts. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 14:56 -0700
Re: No one can do "a perfect forgery" of my posts. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 14:41 -0700
Re: "Path:" is the only line that's never fake. Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 08:54 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-14 11:26 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-14 09:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-14 18:53 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-14 13:27 -0700
Re: slrn is trembling Melzzzzz <Melzzzzz@zzzzz.com> - 2018-10-14 20:55 +0000
Pipelining "Article" commands. Jeff-Relf.Me @. - 2018-10-14 14:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-15 13:49 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 07:34 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-15 21:54 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 19:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 21:51 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 03:16 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-15 20:33 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-16 10:26 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 07:44 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 15:17 +0000
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-16 12:26 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-16 09:46 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 10:21 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-16 14:00 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-16 16:49 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 12:35 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 20:44 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 14:09 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 14:50 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 22:32 +0000
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-04 23:03 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 16:28 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 16:54 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:02 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:20 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:31 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:32 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:54 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:02 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:07 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 18:11 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:48 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 02:06 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:01 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 15:35 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:37 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 08:53 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:57 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:17 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:19 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 09:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 16:19 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:23 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 16:40 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 09:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 17:39 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 11:07 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 18:14 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 11:41 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 18:52 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 08:46 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:29 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:48 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 11:51 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-06 12:19 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-06 15:34 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-08 21:27 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-08 17:49 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-08 18:13 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 07:31 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 13:45 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 07:55 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 15:55 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 09:20 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 09:39 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-09 09:41 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 10:23 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 23:38 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-09 17:47 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-09 11:09 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-10 09:04 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 13:56 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-11 05:02 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:33 -0400
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-10 19:21 +0000
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-12 20:46 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 23:50 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-08 18:44 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-06 14:08 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 11:32 -0700
Re: slrn is trembling "F. Russell" <fr@random.info> - 2018-10-05 00:32 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:46 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:55 +0000
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:17 +0000
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-05 06:59 -0500
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 05:14 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 12:48 +0000
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-05 11:17 -0500
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 13:35 -0400
Re: slrn is trembling Silicon Vee <siliconvee1945@yahoo.com> - 2018-10-11 01:36 +0200
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:07 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 00:12 +0000
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-04 17:21 -0700
Re: slrn is trembling SilverSlimer <silver@slim.er> - 2018-10-05 12:01 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 05:30 -0700
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 08:13 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-05 12:50 -0400
Re: slrn is trembling Steve Carroll <fretwizzer@gmail.com> - 2018-10-05 10:35 -0700
Re: slrn is trembling chrisv <chrisv@nospam.invalid> - 2018-10-10 08:51 -0500
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 07:09 -0700
Re: slrn is trembling Sam O'Nella <poison@donteatit.net> - 2018-10-05 16:43 -0400
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:34 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 07:56 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-10 12:38 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-11 02:12 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-12 12:27 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 12:42 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 21:47 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-14 03:28 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-12 20:45 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-05 15:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 15:41 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 19:21 -0700
Re: slrn is trembling Snit <usenet@gallopinginsanity.com> - 2018-10-05 13:44 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-05 17:48 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 17:15 -0700
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 16:13 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 20:47 -0400
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 01:09 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 18:18 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-04 21:38 -0400
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 19:04 -0700
Re: slrn is trembling owl <owl@rooftop.invalid> - 2018-10-05 02:05 +0000
Re: slrn is trembling Steve Carroll <frelwizzen@gmail.com> - 2018-10-04 19:35 -0700
Re: slrn is trembling DFS <nospam@nospam.com> - 2018-10-10 10:36 -0400
Page 5 of 8 — ← Prev page 1 2 3 4 [5] 6 7 8 Next page →
| From | owl <owl@rooftop.invalid> |
|---|---|
| Date | 2018-10-05 18:14 +0000 |
| Message-ID | <9z0b803ga.3gh@rooftop.invalid> |
| In reply to | #470830 |
Steve Carroll <fretwizzer@gmail.com> wrote: > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote: > > (snip) > >> You're sending that "--START/END OF POST--" to the server. > > Again... DOH! (lifts track shoe off of dick) > >> It doesn't >> know what you're asking. There are delimiters in the article file >> already. A '.' on a line by itself marks the end of an article. > > I saw those. They can never be carried over from an actual line of text? > IOW, are they 100% reliable for delineation? > Yes. A lone '.' is how you end a post when you talk to the server. https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3 >> So you can grep for that: '^\.$', but ^From is just easier to type. >> You can also grep for the last article id number (or loop the whole >> sequence), which will be in the Xref header. > > I'm not yet sure what I'm gonna do here as I have to shoot these > various pieces of data into multiple tables. Remember, you're talking > to a guy who didn't even realize he was sending the above text to the > server (I really gotta start looking at this stuff ;) > >> > And twice each time. BTW, if I remove that dash you suggested, the >> > script won't run. Is the Python nntplib really *that* slow? We're talking >> > 150 seconds compared to 5 for this shell script. >> >> I don't know. > > What else could it be (I've run both a number of times, the speeds are > always roughly what I've reported)? It's probably faster to even do the > next step in a shell script but I really do want to learn Python and DFS > has helped me a *lot* with it and MySQL so I'll stay on that path. I may, > after I get that happening, try it in the shell (I assume there's a way to > do it). In any event, thanks for all the help (and to you, too, DFS). No problem.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-05 11:41 -0700 |
| Message-ID | <fe735165-5f6e-4404-94f5-98a58e6faaea@googlegroups.com> |
| In reply to | #470831 |
On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote:
> Steve Carroll <fretwizzer@gmail.com> wrote:
> > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote:
> >
> > (snip)
> >
> >> You're sending that "--START/END OF POST--" to the server.
> >
> > Again... DOH! (lifts track shoe off of dick)
> >
> >> It doesn't
> >> know what you're asking. There are delimiters in the article file
> >> already. A '.' on a line by itself marks the end of an article.
> >
> > I saw those. They can never be carried over from an actual line of text?
> > IOW, are they 100% reliable for delineation?
> >
>
> Yes. A lone '.' is how you end a post when you talk to the server.
> https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3
Good link.
FYI: I just grabbed 101 posts (705500 - 705600), when I search...
^\.
... I get a count of 106! When I search on...
^220\s\d{6}
... I get the expected 101. I've run these in both Visual Studio Code and
grep in the terminal... same result.
[toc] | [prev] | [next] | [standalone]
| From | owl <owl@rooftop.invalid> |
|---|---|
| Date | 2018-10-05 18:52 +0000 |
| Message-ID | <8z0b803ag.hy93@rooftop.invalid> |
| In reply to | #470834 |
Steve Carroll <fretwizzer@gmail.com> wrote:
> On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote:
>> Steve Carroll <fretwizzer@gmail.com> wrote:
>> > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote:
>> >
>> > (snip)
>> >
>> >> You're sending that "--START/END OF POST--" to the server.
>> >
>> > Again... DOH! (lifts track shoe off of dick)
>> >
>> >> It doesn't
>> >> know what you're asking. There are delimiters in the article file
>> >> already. A '.' on a line by itself marks the end of an article.
>> >
>> > I saw those. They can never be carried over from an actual line of text?
>> > IOW, are they 100% reliable for delineation?
>> >
>>
>> Yes. A lone '.' is how you end a post when you talk to the server.
>> https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3
>
> Good link.
>
> FYI: I just grabbed 101 posts (705500 - 705600), when I search...
>
> ^\.
>
> ... I get a count of 106! When I search on...
>
> ^220\s\d{6}
>
> ... I get the expected 101. I've run these in both Visual Studio Code and
> grep in the terminal... same result.
>
The regex should be '^\.$'
You need an end of line right after the '.'
Without the $ you match every line that starts with a '.'
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-06 08:46 -0700 |
| Message-ID | <519ed4ec-62e5-45a9-a73c-0eff8421c959@googlegroups.com> |
| In reply to | #470838 |
On Friday, October 5, 2018 at 12:52:48 PM UTC-6, owl wrote:
> Steve Carroll <fretwizzer@gmail.com> wrote:
> > On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote:
> >> Steve Carroll <fretwizzer@gmail.com> wrote:
> >> > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote:
> >> >
> >> > (snip)
> >> >
> >> >> You're sending that "--START/END OF POST--" to the server.
> >> >
> >> > Again... DOH! (lifts track shoe off of dick)
> >> >
> >> >> It doesn't
> >> >> know what you're asking. There are delimiters in the article file
> >> >> already. A '.' on a line by itself marks the end of an article.
> >> >
> >> > I saw those. They can never be carried over from an actual line of text?
> >> > IOW, are they 100% reliable for delineation?
> >> >
> >>
> >> Yes. A lone '.' is how you end a post when you talk to the server.
> >> https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3
> >
> > Good link.
> >
> > FYI: I just grabbed 101 posts (705500 - 705600), when I search...
> >
> > ^\.
> >
> > ... I get a count of 106! When I search on...
> >
> > ^220\s\d{6}
> >
> > ... I get the expected 101. I've run these in both Visual Studio Code and
> > grep in the terminal... same result.
> >
>
> The regex should be '^\.$'
> You need an end of line right after the '.'
> Without the $ you match every line that starts with a '.'
LOL! I know better than this. I think I'm trying to spread myself too thin.
I was working on C and DFS came along with all his fancy DB stuff and
convinced me to pick up where I left off (which was at the beginning,
really) so I started in with MySQL. I spent a minute there and pivoted to
Python. Then you pop in with a shell script (that I didn't pay enough
attention to) and I spent part of yesterday messing with awk. For some
reason, regex, even obvious stuff like this, is among the first things I'm
unable to recall.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-06 11:29 -0700 |
| Message-ID | <6c6e4b97-a5a8-41b7-94d2-b9d6eeb2714d@googlegroups.com> |
| In reply to | #470831 |
On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote: > Steve Carroll <fretwizzer@gmail.com> wrote: > > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote: > > > > (snip) > > > >> You're sending that "--START/END OF POST--" to the server. > > > > Again... DOH! (lifts track shoe off of dick) > > > >> It doesn't > >> know what you're asking. There are delimiters in the article file > >> already. A '.' on a line by itself marks the end of an article. > > > > I saw those. They can never be carried over from an actual line of text? > > IOW, are they 100% reliable for delineation? > > > > Yes. A lone '.' is how you end a post when you talk to the server. > https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3 Off topic, if I can pick your brain again... and I know I said I was gonna do this in Python first but you know how it is ;) I had an issue where I had to use sudo to login to mysql, I seem to have fixed it with by updating the plugin to... SET plugin = 'mysql_native_password' ... and flushing privileges. Anyway, to the point... shouldn't I be able to use the following in a shell script to login to mysql with: mysql -h 'localhost' -u 'user_name' -p 'pass_word' -D 'database_name' -t 'table_name' -e 'SELECT * FROM table_name' When I try it (and variations) I get... mysql Ver 14.14 Distrib 5.7.18, for macos10.12 (x86_64) using EditLine wrapper Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Usage: mysql [OPTIONS] [database] ... and the list of available options which I won't include here. Have you ever successfully connected to a MySQL db with a shell script? Seems I can't get it to accept the password in the script. If I remove it from the code I get prompted for it and when I put it in I do get logged in but now I'm at the mysql prompt. Do I write the remainder of the script from within mysql (it let's you have shell access)? Do I need to use read -p (which I've unsuccessfully tried) and just include the password that way? I need to detect when that mysql prompt comes up, too. I've only ever used connectors (Python, Php, etc.) I've never done this from a shell script. If you have *any* info you'd like to impart I'd appreciate it 'cuz this is something I've always wanted to try.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-06 11:48 -0700 |
| Message-ID | <620c44fc-c1e9-45de-b4ab-98e64b42450e@googlegroups.com> |
| In reply to | #470980 |
On Saturday, October 6, 2018 at 12:29:22 PM UTC-6, Steve Carroll wrote: > On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote: > > Steve Carroll <fretwizzer@gmail.com> wrote: > > > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote: > > > > > > (snip) > > > > > >> You're sending that "--START/END OF POST--" to the server. > > > > > > Again... DOH! (lifts track shoe off of dick) > > > > > >> It doesn't > > >> know what you're asking. There are delimiters in the article file > > >> already. A '.' on a line by itself marks the end of an article. > > > > > > I saw those. They can never be carried over from an actual line of text? > > > IOW, are they 100% reliable for delineation? > > > > > > > Yes. A lone '.' is how you end a post when you talk to the server. > > https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3 > > > Off topic, if I can pick your brain again... and I know I said I was gonna > do this in Python first but you know how it is ;) > > I had an issue where I had to use sudo to login to mysql, I seem to > have fixed it with by updating the plugin to... > > SET plugin = 'mysql_native_password' > > ... and flushing privileges. Anyway, to the point... shouldn't I be able to > use the following in a shell script to login to mysql with: > > mysql -h 'localhost' -u 'user_name' -p 'pass_word' -D 'database_name' -t 'table_name' -e 'SELECT * FROM table_name' > > When I try it (and variations) I get... > > mysql Ver 14.14 Distrib 5.7.18, for macos10.12 (x86_64) using EditLine wrapper > Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. > > Oracle is a registered trademark of Oracle Corporation and/or its > affiliates. Other names may be trademarks of their respective > owners. > > Usage: mysql [OPTIONS] [database] > > ... and the list of available options which I won't include here. Have > you ever successfully connected to a MySQL db with a shell script? > > Seems I can't get it to accept the password in the script. If I remove it > from the code I get prompted for it and when I put it in I do get logged > in but now I'm at the mysql prompt. Do I write the remainder of the > script from within mysql (it let's you have shell access)? Do I need > to use read -p (which I've unsuccessfully tried) and just include the > password that way? I need to detect when that mysql prompt comes > up, too. I've only ever used connectors (Python, Php, etc.) I've > never done this from a shell script. If you have *any* info you'd like > to impart I'd appreciate it 'cuz this is something I've always wanted > to try. The following happens in the terminal, when I run: mysql -u 'user_name' -p 'password' or mysql -u 'user_name' --password 'password' I get: ERROR 1049 (42000): Unknown database 'password' However, if I run... mysql -u 'user_name' -p ... and then type in the password at the prompt I get in (same thing in the script, as expected). Is there a config option somewhere that I need to set for mysql?
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-06 11:51 -0700 |
| Message-ID | <ca5bdb47-04ad-4c6c-8b34-993adb6eff28@googlegroups.com> |
| In reply to | #470983 |
On Saturday, October 6, 2018 at 12:48:12 PM UTC-6, Steve Carroll wrote: > On Saturday, October 6, 2018 at 12:29:22 PM UTC-6, Steve Carroll wrote: > > On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote: > > > Steve Carroll <fretwizzer@gmail.com> wrote: > > > > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote: > > > > > > > > (snip) > > > > > > > >> You're sending that "--START/END OF POST--" to the server. > > > > > > > > Again... DOH! (lifts track shoe off of dick) > > > > > > > >> It doesn't > > > >> know what you're asking. There are delimiters in the article file > > > >> already. A '.' on a line by itself marks the end of an article. > > > > > > > > I saw those. They can never be carried over from an actual line of text? > > > > IOW, are they 100% reliable for delineation? > > > > > > > > > > Yes. A lone '.' is how you end a post when you talk to the server. > > > https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3 > > > > > > Off topic, if I can pick your brain again... and I know I said I was gonna > > do this in Python first but you know how it is ;) > > > > I had an issue where I had to use sudo to login to mysql, I seem to > > have fixed it with by updating the plugin to... > > > > SET plugin = 'mysql_native_password' > > > > ... and flushing privileges. Anyway, to the point... shouldn't I be able to > > use the following in a shell script to login to mysql with: > > > > mysql -h 'localhost' -u 'user_name' -p 'pass_word' -D 'database_name' -t 'table_name' -e 'SELECT * FROM table_name' > > > > When I try it (and variations) I get... > > > > mysql Ver 14.14 Distrib 5.7.18, for macos10.12 (x86_64) using EditLine wrapper > > Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. > > > > Oracle is a registered trademark of Oracle Corporation and/or its > > affiliates. Other names may be trademarks of their respective > > owners. > > > > Usage: mysql [OPTIONS] [database] > > > > ... and the list of available options which I won't include here. Have > > you ever successfully connected to a MySQL db with a shell script? > > > > Seems I can't get it to accept the password in the script. If I remove it > > from the code I get prompted for it and when I put it in I do get logged > > in but now I'm at the mysql prompt. Do I write the remainder of the > > script from within mysql (it let's you have shell access)? Do I need > > to use read -p (which I've unsuccessfully tried) and just include the > > password that way? I need to detect when that mysql prompt comes > > up, too. I've only ever used connectors (Python, Php, etc.) I've > > never done this from a shell script. If you have *any* info you'd like > > to impart I'd appreciate it 'cuz this is something I've always wanted > > to try. > > The following happens in the terminal, when I run: > > mysql -u 'user_name' -p 'password' > > or > > mysql -u 'user_name' --password 'password' > > I get: > ERROR 1049 (42000): Unknown database 'password' > > > However, if I run... > > mysql -u 'user_name' -p > > ... and then type in the password at the prompt I get in (same thing > in the script, as expected). Is there a config option somewhere that > I need to set for mysql? I found out you can't have any spaces between -p and the password.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <frelwizzen@gmail.com> |
|---|---|
| Date | 2018-10-06 12:19 -0700 |
| Message-ID | <470141ab-7e64-41cc-a617-f42682bc24b7@googlegroups.com> |
| In reply to | #470984 |
On Saturday, October 6, 2018 at 11:51:15 AM UTC-7, Steve Carroll wrote: > On Saturday, October 6, 2018 at 12:48:12 PM UTC-6, Steve Carroll wrote: > > On Saturday, October 6, 2018 at 12:29:22 PM UTC-6, Steve Carroll wrote: > > > On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote: > > > > Steve Carroll <fretwizzer@gmail.com> wrote: > > > > > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote: > > > > > > > > > > (snip) > > > > > > > > > >> You're sending that "--START/END OF POST--" to the server. > > > > > > > > > > Again... DOH! (lifts track shoe off of dick) > > > > > > > > > >> It doesn't > > > > >> know what you're asking. There are delimiters in the article file > > > > >> already. A '.' on a line by itself marks the end of an article. > > > > > > > > > > I saw those. They can never be carried over from an actual line of text? > > > > > IOW, are they 100% reliable for delineation? > > > > > > > > > > > > > Yes. A lone '.' is how you end a post when you talk to the server. > > > > https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3 > > > > > > > > > Off topic, if I can pick your brain again... and I know I said I was gonna > > > do this in Python first but you know how it is ;) > > > > > > I had an issue where I had to use sudo to login to mysql, I seem to > > > have fixed it with by updating the plugin to... > > > > > > SET plugin = 'mysql_native_password' > > > > > > ... and flushing privileges. Anyway, to the point... shouldn't I be able to > > > use the following in a shell script to login to mysql with: > > > > > > mysql -h 'localhost' -u 'user_name' -p 'pass_word' -D 'database_name' -t 'table_name' -e 'SELECT * FROM table_name' > > > > > > When I try it (and variations) I get... > > > > > > mysql Ver 14.14 Distrib 5.7.18, for macos10.12 (x86_64) using EditLine wrapper > > > Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. > > > > > > Oracle is a registered trademark of Oracle Corporation and/or its > > > affiliates. Other names may be trademarks of their respective > > > owners. > > > > > > Usage: mysql [OPTIONS] [database] > > > > > > ... and the list of available options which I won't include here. Have > > > you ever successfully connected to a MySQL db with a shell script? > > > > > > Seems I can't get it to accept the password in the script. If I remove it > > > from the code I get prompted for it and when I put it in I do get logged > > > in but now I'm at the mysql prompt. Do I write the remainder of the > > > script from within mysql (it let's you have shell access)? Do I need > > > to use read -p (which I've unsuccessfully tried) and just include the > > > password that way? I need to detect when that mysql prompt comes > > > up, too. I've only ever used connectors (Python, Php, etc.) I've > > > never done this from a shell script. If you have *any* info you'd like > > > to impart I'd appreciate it 'cuz this is something I've always wanted > > > to try. > > > > The following happens in the terminal, when I run: > > > > mysql -u 'user_name' -p 'password' > > > > or > > > > mysql -u 'user_name' --password 'password' > > > > I get: > > ERROR 1049 (42000): Unknown database 'password' > > > > > > However, if I run... > > > > mysql -u 'user_name' -p > > > > ... and then type in the password at the prompt I get in (same thing > > in the script, as expected). Is there a config option somewhere that > > I need to set for mysql? > > I found out you can't have any spaces between -p and the password. Both Klaus Schadenfreude and Steven Carroll had their successes and their errors. One played it cool and didn't do anything too horrendous that could not be backed with peer reviews. Linux offers the least choice. It was Steven Carroll who stated that he and his work friends used to commit credit card fraud all the time and it was free entertainment. - Best CMS Solution of 2017 http://www.5z8.info/creditscore_g7f8du_launchexe Jonas Eklundh
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-06 15:34 -0700 |
| Message-ID | <76a6087a-7aa7-4721-a3b1-8b213987883a@googlegroups.com> |
| In reply to | #470984 |
On Saturday, October 6, 2018 at 12:51:15 PM UTC-6, Steve Carroll wrote: > On Saturday, October 6, 2018 at 12:48:12 PM UTC-6, Steve Carroll wrote: > > On Saturday, October 6, 2018 at 12:29:22 PM UTC-6, Steve Carroll wrote: > > > On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote: > > > > Steve Carroll <fretwizzer@gmail.com> wrote: > > > > > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote: > > > > > > > > > > (snip) > > > > > > > > > >> You're sending that "--START/END OF POST--" to the server. > > > > > > > > > > Again... DOH! (lifts track shoe off of dick) > > > > > > > > > >> It doesn't > > > > >> know what you're asking. There are delimiters in the article file > > > > >> already. A '.' on a line by itself marks the end of an article. > > > > > > > > > > I saw those. They can never be carried over from an actual line of text? > > > > > IOW, are they 100% reliable for delineation? > > > > > > > > > > > > > Yes. A lone '.' is how you end a post when you talk to the server. > > > > https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3 > > > > > > > > > Off topic, if I can pick your brain again... and I know I said I was gonna > > > do this in Python first but you know how it is ;) > > > > > > I had an issue where I had to use sudo to login to mysql, I seem to > > > have fixed it with by updating the plugin to... > > > > > > SET plugin = 'mysql_native_password' > > > > > > ... and flushing privileges. Anyway, to the point... shouldn't I be able to > > > use the following in a shell script to login to mysql with: > > > > > > mysql -h 'localhost' -u 'user_name' -p 'pass_word' -D 'database_name' -t 'table_name' -e 'SELECT * FROM table_name' > > > > > > When I try it (and variations) I get... > > > > > > mysql Ver 14.14 Distrib 5.7.18, for macos10.12 (x86_64) using EditLine wrapper > > > Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. > > > > > > Oracle is a registered trademark of Oracle Corporation and/or its > > > affiliates. Other names may be trademarks of their respective > > > owners. > > > > > > Usage: mysql [OPTIONS] [database] > > > > > > ... and the list of available options which I won't include here. Have > > > you ever successfully connected to a MySQL db with a shell script? > > > > > > Seems I can't get it to accept the password in the script. If I remove it > > > from the code I get prompted for it and when I put it in I do get logged > > > in but now I'm at the mysql prompt. Do I write the remainder of the > > > script from within mysql (it let's you have shell access)? Do I need > > > to use read -p (which I've unsuccessfully tried) and just include the > > > password that way? I need to detect when that mysql prompt comes > > > up, too. I've only ever used connectors (Python, Php, etc.) I've > > > never done this from a shell script. If you have *any* info you'd like > > > to impart I'd appreciate it 'cuz this is something I've always wanted > > > to try. > > > > The following happens in the terminal, when I run: > > > > mysql -u 'user_name' -p 'password' > > > > or > > > > mysql -u 'user_name' --password 'password' > > > > I get: > > ERROR 1049 (42000): Unknown database 'password' > > > > > > However, if I run... > > > > mysql -u 'user_name' -p > > > > ... and then type in the password at the prompt I get in (same thing > > in the script, as expected). Is there a config option somewhere that > > I need to set for mysql? > > I found out you can't have any spaces between -p and the password. I had to get away from bash so I went to lunch, when I came back I was just as depressed as I was before ;) BUT... I managed to *finally* piece together this nightmare syntax. The write is not efficient, yet, but it's amazing how fast writes are to the db. I'm just grabbing stuff from a file that has ten posts in it, splitting the output in separate vars and writing: <https://streamable.com/pd6hb> There are probably much better ways to do this (and I mean besides the individual writes) but at least I'm starting to climb the mountain now ;)
[toc] | [prev] | [next] | [standalone]
| From | owl <owl@rooftop.invalid> |
|---|---|
| Date | 2018-10-08 21:27 +0000 |
| Message-ID | <80z08z0b3.agbo3@rooftop.invalid> |
| In reply to | #471011 |
Steve Carroll <fretwizzer@gmail.com> wrote:
> On Saturday, October 6, 2018 at 12:51:15 PM UTC-6, Steve Carroll wrote:
>> On Saturday, October 6, 2018 at 12:48:12 PM UTC-6, Steve Carroll wrote:
>> > On Saturday, October 6, 2018 at 12:29:22 PM UTC-6, Steve Carroll wrote:
>> > > On Friday, October 5, 2018 at 12:14:57 PM UTC-6, owl wrote:
>> > > > Steve Carroll <fretwizzer@gmail.com> wrote:
>> > > > > On Friday, October 5, 2018 at 11:39:47 AM UTC-6, owl wrote:
>> > > > >
>> > > > > (snip)
>> > > > >
>> > > > >> You're sending that "--START/END OF POST--" to the server.
>> > > > >
>> > > > > Again... DOH! (lifts track shoe off of dick)
>> > > > >
>> > > > >> It doesn't
>> > > > >> know what you're asking. There are delimiters in the article file
>> > > > >> already. A '.' on a line by itself marks the end of an article.
>> > > > >
>> > > > > I saw those. They can never be carried over from an actual line of text?
>> > > > > IOW, are they 100% reliable for delineation?
>> > > > >
>> > > >
>> > > > Yes. A lone '.' is how you end a post when you talk to the server.
>> > > > https://tools.ietf.org/html/rfc3977.html#section-6.3.1.3
>> > >
>> > >
>> > > Off topic, if I can pick your brain again... and I know I said I was gonna
>> > > do this in Python first but you know how it is ;)
>> > >
>> > > I had an issue where I had to use sudo to login to mysql, I seem to
>> > > have fixed it with by updating the plugin to...
>> > >
>> > > SET plugin = 'mysql_native_password'
>> > >
>> > > ... and flushing privileges. Anyway, to the point... shouldn't I be able to
>> > > use the following in a shell script to login to mysql with:
>> > >
>> > > mysql -h 'localhost' -u 'user_name' -p 'pass_word' -D 'database_name' -t 'table_name' -e 'SELECT * FROM table_name'
>> > >
>> > > When I try it (and variations) I get...
>> > >
>> > > mysql Ver 14.14 Distrib 5.7.18, for macos10.12 (x86_64) using EditLine wrapper
>> > > Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
>> > >
>> > > Oracle is a registered trademark of Oracle Corporation and/or its
>> > > affiliates. Other names may be trademarks of their respective
>> > > owners.
>> > >
>> > > Usage: mysql [OPTIONS] [database]
>> > >
>> > > ... and the list of available options which I won't include here. Have
>> > > you ever successfully connected to a MySQL db with a shell script?
>> > >
>> > > Seems I can't get it to accept the password in the script. If I remove it
>> > > from the code I get prompted for it and when I put it in I do get logged
>> > > in but now I'm at the mysql prompt. Do I write the remainder of the
>> > > script from within mysql (it let's you have shell access)? Do I need
>> > > to use read -p (which I've unsuccessfully tried) and just include the
>> > > password that way? I need to detect when that mysql prompt comes
>> > > up, too. I've only ever used connectors (Python, Php, etc.) I've
>> > > never done this from a shell script. If you have *any* info you'd like
>> > > to impart I'd appreciate it 'cuz this is something I've always wanted
>> > > to try.
>> >
>> > The following happens in the terminal, when I run:
>> >
>> > mysql -u 'user_name' -p 'password'
>> >
>> > or
>> >
>> > mysql -u 'user_name' --password 'password'
>> >
>> > I get:
>> > ERROR 1049 (42000): Unknown database 'password'
>> >
>> >
>> > However, if I run...
>> >
>> > mysql -u 'user_name' -p
>> >
>> > ... and then type in the password at the prompt I get in (same thing
>> > in the script, as expected). Is there a config option somewhere that
>> > I need to set for mysql?
>>
>> I found out you can't have any spaces between -p and the password.
>
> I had to get away from bash so I went to lunch, when I came back I was
> just as depressed as I was before ;) BUT... I managed to *finally* piece
> together this nightmare syntax. The write is not efficient, yet, but it's
> amazing how fast writes are to the db. I'm just grabbing stuff from a file
> that has ten posts in it, splitting the output in separate vars and writing:
>
> <https://streamable.com/pd6hb>
>
> There are probably much better ways to do this (and I mean besides the
> individual writes) but at least I'm starting to climb the mountain now ;)
I'm far from expert at database stuff, but you could do something like
this:
---------------------------
#!/bin/bash
mysql -pblah --silent -e "create database usenetdb;"
(echo "create table fromtable (id int, content text);"
echo "create table msgidtable (id int, content text);"
echo "create table headers (id int, content text);"
echo "create table body (id int, content text);"
) |mysql -pblah --silent usenetdb
dos2unix -n out newout
csplit -s --suppress-matched newout /^\.$/ {*}
for i in xx*;do
csplit -s -f aa --suppress-matched ${i} /^$/+1
id=$(grep ^220 aa00|awk '{print $2}')
cp aa00 head.${id}
cp aa01 body.${id}
done
rm xx*
rm aa*
(for i in head.*; do
id=$(echo ${i}|sed 's/head\.//g')
fromval=\'$(grep ^From ${i}|cut -f2- -d' ')\'
msgidval=\'$(grep ^Message-ID ${i}|cut -f2- -d' ')\'
headerpath=\'/home/anon/usenet/database/${i}\'
bodypath=\'/home/anon/usenet/database/$(echo ${i}|sed -e 's/head/body/g')\'
echo "insert into fromtable values ( ${id} ,${fromval});"
echo "insert into msgidtable values ( ${id} ,${msgidval});"
echo "insert into headers values ( ${id} , LOAD_FILE( ${headerpath} ) );"
echo "insert into body values ( ${id} , LOAD_FILE( ${bodypath} ) );"
done) |mysql -pblah --silent usenetdb
rm body*
rm head*
rm newout
---------------------------
out is a file of articles created from a bash grab.
I manually trimmed the top and bottom to get rid of connection info clutter,
so it's just articles separated by a '.'
The file has \r\n newlines, so the script runs it through dos2unix and
creates a working copy called "newout".
It then creates a simple database with four tables (headers, body,
msgidtable, fromtable), and parses the "newout" file and does some inserts.
It first splits the file into multiple files named with the article id
number, then deletes them afterward, but you can do the parsing however
you think is easiest.
anon@lowtide:~/usenet/database$ ls
doit out
anon@lowtide:~/usenet/database$ ./doit
dos2unix: converting file out to file newout in Unix format ...
csplit: ‘/^$/+1’: match not found
grep: aa00: No such file or directory
cp: cannot stat ‘aa00’: No such file or directory
anon@lowtide:~/usenet/database$ ls
doit out
anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb
mysql> show tables;
Tables_in_usenetdb
body
fromtable
headers
msgidtable
mysql> select count(*) from body;
count(*)
11
mysql> select count(*) from headers;
count(*)
11
mysql> select count(*) from msgidtable;
count(*)
11
mysql> select count(*) from fromtable;
count(*)
11
mysql> select * from msgidtable;
id content
2365103 <d515d20e-823e-4a43-8651-b76087c88ac9@googlegroups.com>
2365104 <a1f6355f-2628-4bfb-bdd9-beab6f287bf0@googlegroups.com>
2365105 <98fc208f-c1d1-4bc1-b217-d54a55338673@googlegroups.com>
2365106 <c7102860-cb6d-490a-94a6-5ff620b85439@googlegroups.com>
2365107 <e0imrd1ldajcblbgrl0ik3e2s1tcdbapbg@4ax.com>
2365108 <820e7aa5-04ec-48ca-8d96-0ab267229942@googlegroups.com>
2365109 <7d839d14-3a88-4fa5-a5e7-70a263bbaa3c@googlegroups.com>
2365110 <e8f4a698-bc75-492d-a525-c4e80c38096e@googlegroups.com>
2365111 <c23b1b8e-6a5c-4ce5-8dba-2b3adfebd4cf@googlegroups.com>
2365112 <074ddf85-c87a-4669-8d8c-6afc658878ba@googlegroups.com>
2365113 <0f65923b-543c-4e34-811f-f46ba0313768@googlegroups.com>
mysql> select * from body where id=2365110;
id content
2365110 On Sunday, October 7, 2018 at 9:19:14 PM UTC-7, Steve Carroll wrote:\n> He upsets multiple groups of people who are mere civilians, but that's a =\nstuck up jerk for you. What Jonas Eklundh and I care about isn't a factor. =\nYou can say I am Satan himself for all I care. Another successful Linux And=\nroid test. That is if you count utter failure as a success. Klaus Schadenfr=\neude is just thrilled that Jonas Eklundh is on the other end. Imagine if Kl=\naus Schadenfreude walked up to a footstool and delivered the punch line. It=\n wouldn't be rewarding. Perhaps you use it wrong. Do you not fathom the use=\n of https?=20\n>=20\n> Seriously, what lie? I shan't ask Klaus Schadenfreude how any part of Jon=\nas Eklundh's body tastes no matter how often Klaus Schadenfreude kisses it.=\n=20\n>=20\n> --\n> My Snoring Solution\n> https://www.youtube.com/watch?v=3D0ZNxaaKD7-c\n> https://www.youtube.com/watch?v=3DAglvCo3dJ38&feature=3Dyoutu.be\n> Jonas Eklundh\n\n\n\nWhy do you keep repeating yourself?=20\n\nCan you get over it? I bet Steven 'The Moron' Carroll thinks your life was =\nfun.=20\n\nWhat were you doing?=20\n\nThe Underground Marshmallow Person's earliest statement stands proper and c=\norrect.=20\n\n\n\n--\n"You'll notice how quickly he loses interest when everything is about him. =\nHe clearly wants the attention"\nSteve Carroll, making the dumbest comment ever uttered.\n
mysql> quit
anon@lowtide:~/usenet/database$
It's a lot more readable if you don't use the --silent flag, but it
also spits out a bunch of useless crap.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-08 17:49 -0700 |
| Message-ID | <bbd700fd-6dff-4700-adf4-aa725ba29a6b@googlegroups.com> |
| In reply to | #471223 |
On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote:
> Steve Carroll <fretwizzer@gmail.com> wrote:
(snip)
> > There are probably much better ways to do this (and I mean besides the
> > individual writes) but at least I'm starting to climb the mountain now ;)
>
> I'm far from expert at database stuff, but you could do something like
> this:
> ---------------------------
> #!/bin/bash
>
> mysql -pblah --silent -e "create database usenetdb;"
> (echo "create table fromtable (id int, content text);"
> echo "create table msgidtable (id int, content text);"
> echo "create table headers (id int, content text);"
> echo "create table body (id int, content text);"
> ) |mysql -pblah --silent usenetdb
>
> dos2unix -n out newout
I saw dos2unix while researching yesterday, didn't think I needed it but
I got it for the halibut.
> csplit -s --suppress-matched newout /^\.$/ {*}
The --suppress-matched flag doesn't work here and I can't use the
wildcard char in the last arg. This looks like you're trying to delineate
'records' on that line we talked about previously, each into their own
file, yes? Unfortunately, I'm not getting output like that with the above.
> for i in xx*;do
> csplit -s -f aa --suppress-matched ${i} /^$/+1
> id=$(grep ^220 aa00|awk '{print $2}')
So... for every ''xx' file you're splitting each line in it into a new file
(with an 'aa prefix) and this first one is a ref to the article ID, right?
( this syntax and all these options will be the death of me ;)
> cp aa00 head.${id}
> cp aa01 body.${id}
Here, you're copying into (even more!) new files that are labeled
head and body with the appropriate article ID appended so as to
subsequently keep them synced going into the db?
> done
> rm xx*
> rm aa*
At least you remove them ;) You know how many files this would produce
doing 5000 posts?
> (for i in head.*; do
> id=$(echo ${i}|sed 's/head\.//g')
> fromval=\'$(grep ^From ${i}|cut -f2- -d' ')\'
> msgidval=\'$(grep ^Message-ID ${i}|cut -f2- -d' ')\'
> headerpath=\'/home/anon/usenet/database/${i}\'
> bodypath=\'/home/anon/usenet/database/$(echo ${i}|sed -e 's/head/body/g')\'
> echo "insert into fromtable values ( ${id} ,${fromval});"
> echo "insert into msgidtable values ( ${id} ,${msgidval});"
> echo "insert into headers values ( ${id} , LOAD_FILE( ${headerpath} ) );"
> echo "insert into body values ( ${id} , LOAD_FILE( ${bodypath} ) );"
> done) |mysql -pblah --silent usenetdb
>
> rm body*
> rm head*
> rm newout
> ---------------------------
Cool! I tried using an external file (for multiple inserts) from the
command line but it choked on the vars in the file.
I've had issues with cut for some reason, certain flags don't work,
like the above with csplit, I think it's a 'Mack' thing. I gotta get back
to working on this (couldn't do it today) before I forget half the crap
I've researched. I haven't gotten very far yet... anyway, instead of
cutting it all up, I started doing this (which ran in a while loop):
if [[ $line =~ $articleP ]];then
artID=`echo "$line" | cut -b 12-17`
elif [[ $line =~ $fromP ]];then
from=`echo "$line" | cut -b 7-1000`
elif [[ $line =~ $subjectP ]];then
subject=`echo "$line" | cut -b 10-1000`
elif [[ $line =~ $messageIDP ]];then
msgID=`echo "$line" | cut -b 13-1000`
fi
... but I think I did because I'm not used to the 'bash way', which
seems to be churn a buttload of files... I'm used to using RAM. I
was thinking of pushing those into an array at each step above
and figuring out how to do the db write from that (while adding
the body text, as well). It's amazing how fast stuff is in this
environment, a line like this happens in a couple of seconds:
sed -n '/^220/,/^\s*$/p' post_grab_5000 > post_headers_5000
I was trying to use Xref in a body grab (to sync with) but it seems
it may show up more than once in a post (not sure). I tried...
sed -n '/^Xref:/,/^\..*/p' post_grab_5000
... and the number was off slightly. I was thinking about trying to
generate 1 temp file each, for the header and the body, each time
it gets processed and written but I never got that far.
I also tried...
awk '/^\..*/ {exit} 1' post_grab_10 > temp
... with the thought of using it to loop/process but I couldn't figure
out how to do it before I had to leave (and still haven't).
> out is a file of articles created from a bash grab.
> I manually trimmed the top and bottom to get rid of connection info clutter,
> so it's just articles separated by a '.'
> The file has \r\n newlines, so the script runs it through dos2unix and
> creates a working copy called "newout".
>
> It then creates a simple database with four tables (headers, body,
> msgidtable, fromtable), and parses the "newout" file and does some inserts.
> It first splits the file into multiple files named with the article id
> number, then deletes them afterward, but you can do the parsing however
> you think is easiest.
>
> anon@lowtide:~/usenet/database$ ls
> doit out
> anon@lowtide:~/usenet/database$ ./doit
> dos2unix: converting file out to file newout in Unix format ...
> csplit: ‘/^$/+1’: match not found
> grep: aa00: No such file or directory
> cp: cannot stat ‘aa00’: No such file or directory
> anon@lowtide:~/usenet/database$ ls
> doit out
> anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb
> mysql> show tables;
> Tables_in_usenetdb
> body
> fromtable
> headers
> msgidtable
> mysql> select count(*) from body;
> count(*)
> 11
> mysql> select count(*) from headers;
> count(*)
> 11
> mysql> select count(*) from msgidtable;
> count(*)
> 11
> mysql> select count(*) from fromtable;
> count(*)
> 11
> mysql> select * from msgidtable;
> id content
> 2365103 <d515d20e-823e-4a43-8651-b76087c88ac9@googlegroups.com>
> 2365104 <a1f6355f-2628-4bfb-bdd9-beab6f287bf0@googlegroups.com>
> 2365105 <98fc208f-c1d1-4bc1-b217-d54a55338673@googlegroups.com>
> 2365106 <c7102860-cb6d-490a-94a6-5ff620b85439@googlegroups.com>
> 2365107 <e0imrd1ldajcblbgrl0ik3e2s1tcdbapbg@4ax.com>
> 2365108 <820e7aa5-04ec-48ca-8d96-0ab267229942@googlegroups.com>
> 2365109 <7d839d14-3a88-4fa5-a5e7-70a263bbaa3c@googlegroups.com>
> 2365110 <e8f4a698-bc75-492d-a525-c4e80c38096e@googlegroups.com>
> 2365111 <c23b1b8e-6a5c-4ce5-8dba-2b3adfebd4cf@googlegroups.com>
> 2365112 <074ddf85-c87a-4669-8d8c-6afc658878ba@googlegroups.com>
> 2365113 <0f65923b-543c-4e34-811f-f46ba0313768@googlegroups.com>
> mysql> select * from body where id=2365110;
> id content
> 2365110 On Sunday, October 7, 2018 at 9:19:14 PM UTC-7, Steve Carroll wrote:\n> He upsets multiple groups of people who are mere civilians, but that's a =\nstuck up jerk for you. What Jonas Eklundh and I care about isn't a factor. =\nYou can say I am Satan himself for all I care. Another successful Linux And=\nroid test. That is if you count utter failure as a success. Klaus Schadenfr=\neude is just thrilled that Jonas Eklundh is on the other end. Imagine if Kl=\naus Schadenfreude walked up to a footstool and delivered the punch line. It=\n wouldn't be rewarding. Perhaps you use it wrong. Do you not fathom the use=\n of https?=20\n>=20\n> Seriously, what lie? I shan't ask Klaus Schadenfreude how any part of Jon=\nas Eklundh's body tastes no matter how often Klaus Schadenfreude kisses it.=\n=20\n>=20\n> --\n> My Snoring Solution\n> https://www.youtube.com/watch?v=3D0ZNxaaKD7-c\n> https://www.youtube.com/watch?v=3DAglvCo3dJ38&feature=3Dyoutu.be\n> Jonas Eklundh\n\n\n\nWhy do you keep repeating yourself?=20\n\nCan you get over it? I bet Steven 'The Moron' Carroll thinks your life was =\nfun.=20\n\nWhat were you doing?=20\n\nThe Underground Marshmallow Person's earliest statement stands proper and c=\norrect.=20\n\n\n\n--\n"You'll notice how quickly he loses interest when everything is about him. =\nHe clearly wants the attention"\nSteve Carroll, making the dumbest comment ever uttered.\n
> mysql> quit
> anon@lowtide:~/usenet/database$
>
>
> It's a lot more readable if you don't use the --silent flag, but it
> also spits out a bunch of useless crap.
Thanks for all this stuff!
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <frelwizzen@gmail.com> |
|---|---|
| Date | 2018-10-08 18:13 -0700 |
| Message-ID | <bb616e16-7ef3-4159-95b7-e13c5498e358@googlegroups.com> |
| In reply to | #471242 |
On Monday, October 8, 2018 at 5:49:38 PM UTC-7, Steve Carroll wrote:
> On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote:
> > Steve Carroll <fretwizzer@gmail.com> wrote:
>
> (snip)
>
> > > There are probably much better ways to do this (and I mean besides the
> > > individual writes) but at least I'm starting to climb the mountain now ;)
> >
> > I'm far from expert at database stuff, but you could do something like
> > this:
> > ---------------------------
> > #!/bin/bash
> >
> > mysql -pblah --silent -e "create database usenetdb;"
> > (echo "create table fromtable (id int, content text);"
> > echo "create table msgidtable (id int, content text);"
> > echo "create table headers (id int, content text);"
> > echo "create table body (id int, content text);"
> > ) |mysql -pblah --silent usenetdb
> >
> > dos2unix -n out newout
>
> I saw dos2unix while researching yesterday, didn't think I needed it but
> I got it for the halibut.
>
> > csplit -s --suppress-matched newout /^\.$/ {*}
>
> The --suppress-matched flag doesn't work here and I can't use the
> wildcard char in the last arg. This looks like you're trying to delineate
> 'records' on that line we talked about previously, each into their own
> file, yes? Unfortunately, I'm not getting output like that with the above.
>
> > for i in xx*;do
> > csplit -s -f aa --suppress-matched ${i} /^$/+1
> > id=$(grep ^220 aa00|awk '{print $2}')
>
> So... for every ''xx' file you're splitting each line in it into a new file
> (with an 'aa prefix) and this first one is a ref to the article ID, right?
> ( this syntax and all these options will be the death of me ;)
>
> > cp aa00 head.${id}
> > cp aa01 body.${id}
>
> Here, you're copying into (even more!) new files that are labeled
> head and body with the appropriate article ID appended so as to
> subsequently keep them synced going into the db?
>
> > done
> > rm xx*
> > rm aa*
>
> At least you remove them ;) You know how many files this would produce
> doing 5000 posts?
>
> > (for i in head.*; do
> > id=$(echo ${i}|sed 's/head\.//g')
> > fromval=\'$(grep ^From ${i}|cut -f2- -d' ')\'
> > msgidval=\'$(grep ^Message-ID ${i}|cut -f2- -d' ')\'
> > headerpath=\'/home/anon/usenet/database/${i}\'
> > bodypath=\'/home/anon/usenet/database/$(echo ${i}|sed -e 's/head/body/g')\'
> > echo "insert into fromtable values ( ${id} ,${fromval});"
> > echo "insert into msgidtable values ( ${id} ,${msgidval});"
> > echo "insert into headers values ( ${id} , LOAD_FILE( ${headerpath} ) );"
> > echo "insert into body values ( ${id} , LOAD_FILE( ${bodypath} ) );"
> > done) |mysql -pblah --silent usenetdb
> >
> > rm body*
> > rm head*
> > rm newout
> > ---------------------------
>
> Cool! I tried using an external file (for multiple inserts) from the
> command line but it choked on the vars in the file.
>
> I've had issues with cut for some reason, certain flags don't work,
> like the above with csplit, I think it's a 'Mack' thing. I gotta get back
> to working on this (couldn't do it today) before I forget half the crap
> I've researched. I haven't gotten very far yet... anyway, instead of
> cutting it all up, I started doing this (which ran in a while loop):
>
> if [[ $line =~ $articleP ]];then
> artID=`echo "$line" | cut -b 12-17`
> elif [[ $line =~ $fromP ]];then
> from=`echo "$line" | cut -b 7-1000`
> elif [[ $line =~ $subjectP ]];then
> subject=`echo "$line" | cut -b 10-1000`
> elif [[ $line =~ $messageIDP ]];then
> msgID=`echo "$line" | cut -b 13-1000`
> fi
>
> ... but I think I did because I'm not used to the 'bash way', which
> seems to be churn a buttload of files... I'm used to using RAM. I
> was thinking of pushing those into an array at each step above
> and figuring out how to do the db write from that (while adding
> the body text, as well). It's amazing how fast stuff is in this
> environment, a line like this happens in a couple of seconds:
>
> sed -n '/^220/,/^\s*$/p' post_grab_5000 > post_headers_5000
>
> I was trying to use Xref in a body grab (to sync with) but it seems
> it may show up more than once in a post (not sure). I tried...
>
> sed -n '/^Xref:/,/^\..*/p' post_grab_5000
>
> ... and the number was off slightly. I was thinking about trying to
> generate 1 temp file each, for the header and the body, each time
> it gets processed and written but I never got that far.
>
> I also tried...
>
> awk '/^\..*/ {exit} 1' post_grab_10 > temp
>
> ... with the thought of using it to loop/process but I couldn't figure
> out how to do it before I had to leave (and still haven't).
>
> > out is a file of articles created from a bash grab.
> > I manually trimmed the top and bottom to get rid of connection info clutter,
> > so it's just articles separated by a '.'
> > The file has \r\n newlines, so the script runs it through dos2unix and
> > creates a working copy called "newout".
> >
> > It then creates a simple database with four tables (headers, body,
> > msgidtable, fromtable), and parses the "newout" file and does some inserts.
> > It first splits the file into multiple files named with the article id
> > number, then deletes them afterward, but you can do the parsing however
> > you think is easiest.
> >
> > anon@lowtide:~/usenet/database$ ls
> > doit out
> > anon@lowtide:~/usenet/database$ ./doit
> > dos2unix: converting file out to file newout in Unix format ...
> > csplit: ‘/^$/+1’: match not found
> > grep: aa00: No such file or directory
> > cp: cannot stat ‘aa00’: No such file or directory
> > anon@lowtide:~/usenet/database$ ls
> > doit out
> > anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb
> > mysql> show tables;
> > Tables_in_usenetdb
> > body
> > fromtable
> > headers
> > msgidtable
> > mysql> select count(*) from body;
> > count(*)
> > 11
> > mysql> select count(*) from headers;
> > count(*)
> > 11
> > mysql> select count(*) from msgidtable;
> > count(*)
> > 11
> > mysql> select count(*) from fromtable;
> > count(*)
> > 11
> > mysql> select * from msgidtable;
> > id content
> > 2365103 <d515d20e-823e-4a43-8651-b76087c88ac9@googlegroups.com>
> > 2365104 <a1f6355f-2628-4bfb-bdd9-beab6f287bf0@googlegroups.com>
> > 2365105 <98fc208f-c1d1-4bc1-b217-d54a55338673@googlegroups.com>
> > 2365106 <c7102860-cb6d-490a-94a6-5ff620b85439@googlegroups.com>
> > 2365107 <e0imrd1ldajcblbgrl0ik3e2s1tcdbapbg@4ax.com>
> > 2365108 <820e7aa5-04ec-48ca-8d96-0ab267229942@googlegroups.com>
> > 2365109 <7d839d14-3a88-4fa5-a5e7-70a263bbaa3c@googlegroups.com>
> > 2365110 <e8f4a698-bc75-492d-a525-c4e80c38096e@googlegroups.com>
> > 2365111 <c23b1b8e-6a5c-4ce5-8dba-2b3adfebd4cf@googlegroups.com>
> > 2365112 <074ddf85-c87a-4669-8d8c-6afc658878ba@googlegroups.com>
> > 2365113 <0f65923b-543c-4e34-811f-f46ba0313768@googlegroups.com>
> > mysql> select * from body where id=2365110;
> > id content
> > 2365110 On Sunday, October 7, 2018 at 9:19:14 PM UTC-7, Steve Carroll wrote:\n> He upsets multiple groups of people who are mere civilians, but that's a =\nstuck up jerk for you. What Jonas Eklundh and I care about isn't a factor. =\nYou can say I am Satan himself for all I care. Another successful Linux And=\nroid test. That is if you count utter failure as a success. Klaus Schadenfr=\neude is just thrilled that Jonas Eklundh is on the other end. Imagine if Kl=\naus Schadenfreude walked up to a footstool and delivered the punch line. It=\n wouldn't be rewarding. Perhaps you use it wrong. Do you not fathom the use=\n of https?=20\n>=20\n> Seriously, what lie? I shan't ask Klaus Schadenfreude how any part of Jon=\nas Eklundh's body tastes no matter how often Klaus Schadenfreude kisses it.=\n=20\n>=20\n> --\n> My Snoring Solution\n> https://www.youtube.com/watch?v=3D0ZNxaaKD7-c\n> https://www.youtube.com/watch?v=3DAglvCo3dJ38&feature=3Dyoutu.be\n> Jonas Eklundh\n\n\n\nWhy do you keep repeating yourself?=20\n\nCan you get over it? I bet Steven 'The Moron' Carroll thinks your life was =\nfun.=20\n\nWhat were you doing?=20\n\nThe Underground Marshmallow Person's earliest statement stands proper and c=\norrect.=20\n\n\n\n--\n"You'll notice how quickly he loses interest when everything is about him. =\nHe clearly wants the attention"\nSteve Carroll, making the dumbest comment ever uttered.\n
> > mysql> quit
> > anon@lowtide:~/usenet/database$
> >
> >
> > It's a lot more readable if you don't use the --silent flag, but it
> > also spits out a bunch of useless crap.
>
> Thanks for all this stuff!
My uptime is almost sixty-two days and Wayland is still going strong!
Klaus Schadenfreude's computer has more cores than F. Russell's. Klaus Schadenfreude wins. F. Russell loses. Which gives Klaus Schadenfreude better productivity, efficiency, and error-reduction.
You do realize that the massive floods ending up in multiple groups started out in COLA, right?
Was that meant to be clever? Time to blame the herd! You've proven nothing.
--
Do not click this link!
https://www.youtube.com/playlist?list=PLU_evUKollmjeuJy5V5UMotPUs65Q7YM6
http://tmp.gallopinginsanity.com/BilkHelp.html
Jonas Eklundh Communication AB
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <frelwizzen@gmail.com> |
|---|---|
| Date | 2018-10-09 07:31 -0700 |
| Message-ID | <ce63cc0b-7703-4f44-8bfc-1bddcf73910f@googlegroups.com> |
| In reply to | #471244 |
On Monday, October 8, 2018 at 6:13:22 PM UTC-7, Steve Carroll wrote:
> On Monday, October 8, 2018 at 5:49:38 PM UTC-7, Steve Carroll wrote:
> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote:
> > > Steve Carroll <fretwizzer@gmail.com> wrote:
> >
> > (snip)
> >
> > > > There are probably much better ways to do this (and I mean besides the
> > > > individual writes) but at least I'm starting to climb the mountain now ;)
> > >
> > > I'm far from expert at database stuff, but you could do something like
> > > this:
> > > ---------------------------
> > > #!/bin/bash
> > >
> > > mysql -pblah --silent -e "create database usenetdb;"
> > > (echo "create table fromtable (id int, content text);"
> > > echo "create table msgidtable (id int, content text);"
> > > echo "create table headers (id int, content text);"
> > > echo "create table body (id int, content text);"
> > > ) |mysql -pblah --silent usenetdb
> > >
> > > dos2unix -n out newout
> >
> > I saw dos2unix while researching yesterday, didn't think I needed it but
> > I got it for the halibut.
> >
> > > csplit -s --suppress-matched newout /^\.$/ {*}
> >
> > The --suppress-matched flag doesn't work here and I can't use the
> > wildcard char in the last arg. This looks like you're trying to delineate
> > 'records' on that line we talked about previously, each into their own
> > file, yes? Unfortunately, I'm not getting output like that with the above.
> >
> > > for i in xx*;do
> > > csplit -s -f aa --suppress-matched ${i} /^$/+1
> > > id=$(grep ^220 aa00|awk '{print $2}')
> >
> > So... for every ''xx' file you're splitting each line in it into a new file
> > (with an 'aa prefix) and this first one is a ref to the article ID, right?
> > ( this syntax and all these options will be the death of me ;)
> >
> > > cp aa00 head.${id}
> > > cp aa01 body.${id}
> >
> > Here, you're copying into (even more!) new files that are labeled
> > head and body with the appropriate article ID appended so as to
> > subsequently keep them synced going into the db?
> >
> > > done
> > > rm xx*
> > > rm aa*
> >
> > At least you remove them ;) You know how many files this would produce
> > doing 5000 posts?
> >
> > > (for i in head.*; do
> > > id=$(echo ${i}|sed 's/head\.//g')
> > > fromval=\'$(grep ^From ${i}|cut -f2- -d' ')\'
> > > msgidval=\'$(grep ^Message-ID ${i}|cut -f2- -d' ')\'
> > > headerpath=\'/home/anon/usenet/database/${i}\'
> > > bodypath=\'/home/anon/usenet/database/$(echo ${i}|sed -e 's/head/body/g')\'
> > > echo "insert into fromtable values ( ${id} ,${fromval});"
> > > echo "insert into msgidtable values ( ${id} ,${msgidval});"
> > > echo "insert into headers values ( ${id} , LOAD_FILE( ${headerpath} ) );"
> > > echo "insert into body values ( ${id} , LOAD_FILE( ${bodypath} ) );"
> > > done) |mysql -pblah --silent usenetdb
> > >
> > > rm body*
> > > rm head*
> > > rm newout
> > > ---------------------------
> >
> > Cool! I tried using an external file (for multiple inserts) from the
> > command line but it choked on the vars in the file.
> >
> > I've had issues with cut for some reason, certain flags don't work,
> > like the above with csplit, I think it's a 'Mack' thing. I gotta get back
> > to working on this (couldn't do it today) before I forget half the crap
> > I've researched. I haven't gotten very far yet... anyway, instead of
> > cutting it all up, I started doing this (which ran in a while loop):
> >
> > if [[ $line =~ $articleP ]];then
> > artID=`echo "$line" | cut -b 12-17`
> > elif [[ $line =~ $fromP ]];then
> > from=`echo "$line" | cut -b 7-1000`
> > elif [[ $line =~ $subjectP ]];then
> > subject=`echo "$line" | cut -b 10-1000`
> > elif [[ $line =~ $messageIDP ]];then
> > msgID=`echo "$line" | cut -b 13-1000`
> > fi
> >
> > ... but I think I did because I'm not used to the 'bash way', which
> > seems to be churn a buttload of files... I'm used to using RAM. I
> > was thinking of pushing those into an array at each step above
> > and figuring out how to do the db write from that (while adding
> > the body text, as well). It's amazing how fast stuff is in this
> > environment, a line like this happens in a couple of seconds:
> >
> > sed -n '/^220/,/^\s*$/p' post_grab_5000 > post_headers_5000
> >
> > I was trying to use Xref in a body grab (to sync with) but it seems
> > it may show up more than once in a post (not sure). I tried...
> >
> > sed -n '/^Xref:/,/^\..*/p' post_grab_5000
> >
> > ... and the number was off slightly. I was thinking about trying to
> > generate 1 temp file each, for the header and the body, each time
> > it gets processed and written but I never got that far.
> >
> > I also tried...
> >
> > awk '/^\..*/ {exit} 1' post_grab_10 > temp
> >
> > ... with the thought of using it to loop/process but I couldn't figure
> > out how to do it before I had to leave (and still haven't).
> >
> > > out is a file of articles created from a bash grab.
> > > I manually trimmed the top and bottom to get rid of connection info clutter,
> > > so it's just articles separated by a '.'
> > > The file has \r\n newlines, so the script runs it through dos2unix and
> > > creates a working copy called "newout".
> > >
> > > It then creates a simple database with four tables (headers, body,
> > > msgidtable, fromtable), and parses the "newout" file and does some inserts.
> > > It first splits the file into multiple files named with the article id
> > > number, then deletes them afterward, but you can do the parsing however
> > > you think is easiest.
> > >
> > > anon@lowtide:~/usenet/database$ ls
> > > doit out
> > > anon@lowtide:~/usenet/database$ ./doit
> > > dos2unix: converting file out to file newout in Unix format ...
> > > csplit: ‘/^$/+1’: match not found
> > > grep: aa00: No such file or directory
> > > cp: cannot stat ‘aa00’: No such file or directory
> > > anon@lowtide:~/usenet/database$ ls
> > > doit out
> > > anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb
> > > mysql> show tables;
> > > Tables_in_usenetdb
> > > body
> > > fromtable
> > > headers
> > > msgidtable
> > > mysql> select count(*) from body;
> > > count(*)
> > > 11
> > > mysql> select count(*) from headers;
> > > count(*)
> > > 11
> > > mysql> select count(*) from msgidtable;
> > > count(*)
> > > 11
> > > mysql> select count(*) from fromtable;
> > > count(*)
> > > 11
> > > mysql> select * from msgidtable;
> > > id content
> > > 2365103 <d515d20e-823e-4a43-8651-b76087c88ac9@googlegroups.com>
> > > 2365104 <a1f6355f-2628-4bfb-bdd9-beab6f287bf0@googlegroups.com>
> > > 2365105 <98fc208f-c1d1-4bc1-b217-d54a55338673@googlegroups.com>
> > > 2365106 <c7102860-cb6d-490a-94a6-5ff620b85439@googlegroups.com>
> > > 2365107 <e0imrd1ldajcblbgrl0ik3e2s1tcdbapbg@4ax.com>
> > > 2365108 <820e7aa5-04ec-48ca-8d96-0ab267229942@googlegroups.com>
> > > 2365109 <7d839d14-3a88-4fa5-a5e7-70a263bbaa3c@googlegroups.com>
> > > 2365110 <e8f4a698-bc75-492d-a525-c4e80c38096e@googlegroups.com>
> > > 2365111 <c23b1b8e-6a5c-4ce5-8dba-2b3adfebd4cf@googlegroups.com>
> > > 2365112 <074ddf85-c87a-4669-8d8c-6afc658878ba@googlegroups.com>
> > > 2365113 <0f65923b-543c-4e34-811f-f46ba0313768@googlegroups.com>
> > > mysql> select * from body where id=2365110;
> > > id content
> > > 2365110 On Sunday, October 7, 2018 at 9:19:14 PM UTC-7, Steve Carroll wrote:\n> He upsets multiple groups of people who are mere civilians, but that's a =\nstuck up jerk for you. What Jonas Eklundh and I care about isn't a factor. =\nYou can say I am Satan himself for all I care. Another successful Linux And=\nroid test. That is if you count utter failure as a success. Klaus Schadenfr=\neude is just thrilled that Jonas Eklundh is on the other end. Imagine if Kl=\naus Schadenfreude walked up to a footstool and delivered the punch line. It=\n wouldn't be rewarding. Perhaps you use it wrong. Do you not fathom the use=\n of https?=20\n>=20\n> Seriously, what lie? I shan't ask Klaus Schadenfreude how any part of Jon=\nas Eklundh's body tastes no matter how often Klaus Schadenfreude kisses it.=\n=20\n>=20\n> --\n> My Snoring Solution\n> https://www.youtube.com/watch?v=3D0ZNxaaKD7-c\n> https://www.youtube.com/watch?v=3DAglvCo3dJ38&feature=3Dyoutu.be\n> Jonas Eklundh\n\n\n\nWhy do you keep repeating yourself?=20\n\nCan you get over it? I bet Steven 'The Moron' Carroll thinks your life was =\nfun.=20\n\nWhat were you doing?=20\n\nThe Underground Marshmallow Person's earliest statement stands proper and c=\norrect.=20\n\n\n\n--\n"You'll notice how quickly he loses interest when everything is about him. =\nHe clearly wants the attention"\nSteve Carroll, making the dumbest comment ever uttered.\n
> > > mysql> quit
> > > anon@lowtide:~/usenet/database$
> > >
> > >
> > > It's a lot more readable if you don't use the --silent flag, but it
> > > also spits out a bunch of useless crap.
> >
> > Thanks for all this stuff!
>
>
>
> My uptime is almost sixty-two days and Wayland is still going strong!
>
> Klaus Schadenfreude's computer has more cores than F. Russell's. Klaus Schadenfreude wins. F. Russell loses. Which gives Klaus Schadenfreude better productivity, efficiency, and error-reduction.
>
> You do realize that the massive floods ending up in multiple groups started out in COLA, right?
>
> Was that meant to be clever? Time to blame the herd! You've proven nothing.
>
> --
> Do not click this link!
> https://www.youtube.com/playlist?list=PLU_evUKollmjeuJy5V5UMotPUs65Q7YM6
> http://tmp.gallopinginsanity.com/BilkHelp.html
> Jonas Eklundh Communication AB
It was Klaus Schadenfreude who forged me and did not even try to hide it.
With no evidence at all, as is usual for Klaus Schadenfreude. Klaus Schadenfreude's passed old text through Google translate with (I am betting) Markova chains to regurgitate posts which are in the form of ones from Satan's. Klaus Schadenfreude is an idiot... advocates have plonked his socks. What he really should use is a bot that keeps switching names, then flood posts have a shot at being one quarter as repetitive as he is.
Lines of text containing quotes you somehow recognize but can not place.
What it looks like can be skinned and not tied to its functionality, especially if it's approved by RMS. On and on. You're again trying to act like you know as much as Satan. You're the village idiot and your many poorly done tasks show this.
--
Puppy Videos!!
https://youtu.be/u4xD43Khhkw
https://youtu.be/VxLiH-x0aeY
http://www.5z8.info/facebook-of-sex_w4l4wg_bomb
Jonas Eklundh
[toc] | [prev] | [next] | [standalone]
| From | owl <owl@rooftop.invalid> |
|---|---|
| Date | 2018-10-09 13:45 +0000 |
| Message-ID | <pk9a99ba3.l4a@rooftop.invalid> |
| In reply to | #471242 |
Steve Carroll <fretwizzer@gmail.com> wrote:
> On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote:
>> Steve Carroll <fretwizzer@gmail.com> wrote:
>
> (snip)
>
>> > There are probably much better ways to do this (and I mean besides the
>> > individual writes) but at least I'm starting to climb the mountain now ;)
>>
>> I'm far from expert at database stuff, but you could do something like
>> this:
>> ---------------------------
>> #!/bin/bash
>>
>> mysql -pblah --silent -e "create database usenetdb;"
>> (echo "create table fromtable (id int, content text);"
>> echo "create table msgidtable (id int, content text);"
>> echo "create table headers (id int, content text);"
>> echo "create table body (id int, content text);"
>> ) |mysql -pblah --silent usenetdb
>>
>> dos2unix -n out newout
>
> I saw dos2unix while researching yesterday, didn't think I needed it but
> I got it for the halibut.
>
The CRLF fucks with the regex matching. I didn't want to bother
with them, so I just get rid of them.
>> csplit -s --suppress-matched newout /^\.$/ {*}
>
> The --suppress-matched flag doesn't work here and I can't use the
> wildcard char in the last arg. This looks like you're trying to delineate
> 'records' on that line we talked about previously, each into their own
> file, yes? Unfortunately, I'm not getting output like that with the above.
>
>> for i in xx*;do
>> csplit -s -f aa --suppress-matched ${i} /^$/+1
>> id=$(grep ^220 aa00|awk '{print $2}')
>
> So... for every ''xx' file you're splitting each line in it into a new file
> (with an 'aa prefix) and this first one is a ref to the article ID, right?
> ( this syntax and all these options will be the death of me ;)
>
No, it splits every xx file into two files, aa0 and aa1.
>> cp aa00 head.${id}
>> cp aa01 body.${id}
>
> Here, you're copying into (even more!) new files that are labeled
> head and body with the appropriate article ID appended so as to
> subsequently keep them synced going into the db?
>
Yes.
>> done
>> rm xx*
>> rm aa*
>
> At least you remove them ;) You know how many files this would produce
> doing 5000 posts?
>
About 15000, I think.
>> It's a lot more readable if you don't use the --silent flag, but it
>> also spits out a bunch of useless crap.
>
> Thanks for all this stuff!
It doesn't work right for a large number of posts, and there's a problem
with the csplit regex, so I'm scrapping the whole shell approach. Too
much labor to make it not messy, and I don't trust bash arrays for
large amounts of big data.
I'm working on one in C, with everything in RAM. With a 1000-post file,
it does inserts to the fromtable and headers table, and takes about
1.5 minutes for the 2000 inserts. That doesn't seem very good to me.
The inserts are a real performance bottleneck. How do JS and Python
do for insert time?
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-09 07:55 -0700 |
| Message-ID | <a3ce18b3-1ab8-4a63-bfd9-c8c01a3ed760@googlegroups.com> |
| In reply to | #471289 |
On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote:
> Steve Carroll <fretwizzer@gmail.com> wrote:
> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote:
> >> Steve Carroll <fretwizzer@gmail.com> wrote:
> >
> > (snip)
> >
> >> > There are probably much better ways to do this (and I mean besides the
> >> > individual writes) but at least I'm starting to climb the mountain now ;)
> >>
> >> I'm far from expert at database stuff, but you could do something like
> >> this:
> >> ---------------------------
> >> #!/bin/bash
> >>
> >> mysql -pblah --silent -e "create database usenetdb;"
> >> (echo "create table fromtable (id int, content text);"
> >> echo "create table msgidtable (id int, content text);"
> >> echo "create table headers (id int, content text);"
> >> echo "create table body (id int, content text);"
> >> ) |mysql -pblah --silent usenetdb
> >>
> >> dos2unix -n out newout
> >
> > I saw dos2unix while researching yesterday, didn't think I needed it but
> > I got it for the halibut.
> >
>
> The CRLF fucks with the regex matching. I didn't want to bother
> with them, so I just get rid of them.
>
> >> csplit -s --suppress-matched newout /^\.$/ {*}
> >
> > The --suppress-matched flag doesn't work here and I can't use the
> > wildcard char in the last arg. This looks like you're trying to delineate
> > 'records' on that line we talked about previously, each into their own
> > file, yes? Unfortunately, I'm not getting output like that with the above.
> >
> >> for i in xx*;do
> >> csplit -s -f aa --suppress-matched ${i} /^$/+1
> >> id=$(grep ^220 aa00|awk '{print $2}')
> >
> > So... for every ''xx' file you're splitting each line in it into a new file
> > (with an 'aa prefix) and this first one is a ref to the article ID, right?
> > ( this syntax and all these options will be the death of me ;)
> >
>
>
> No, it splits every xx file into two files, aa0 and aa1.
I figured that was your goal... but when I tried it all I got was a new
file per line until it hit the last number (I couldn't use the wildcard),
then the rest of the entire file went into the final new file. I still don't
understand why, though. What's the pattern in csplit do? And what's
the '+1' for after it?
> >> cp aa00 head.${id}
> >> cp aa01 body.${id}
> >
> > Here, you're copying into (even more!) new files that are labeled
> > head and body with the appropriate article ID appended so as to
> > subsequently keep them synced going into the db?
> >
>
> Yes.
>
> >> done
> >> rm xx*
> >> rm aa*
> >
> > At least you remove them ;) You know how many files this would produce
> > doing 5000 posts?
> >
>
> About 15000, I think.
>
>
> >> It's a lot more readable if you don't use the --silent flag, but it
> >> also spits out a bunch of useless crap.
> >
> > Thanks for all this stuff!
>
>
> It doesn't work right for a large number of posts, and there's a problem
> with the csplit regex, so I'm scrapping the whole shell approach. Too
> much labor to make it not messy, and I don't trust bash arrays for
> large amounts of big data.
I was thinking of refreshing the array with the processing and writing
of each record but I haven't yet figured out how while keeping things
in sync. What's the problem with large bash arrays?
> I'm working on one in C, with everything in RAM.
To me that makes more sense (why pound a drive when you have RAM
which is faster anyway?). I was messing with C before I started with this
stuff again but, from what little I saw, the syntax wasn't as grueling as
Bash. I was on step one with C... I didn't get to the point where I was
using system() or anything like that so I'd have to figure out how to get
all these little utility apps (awk, sed, cut, etc.) working in it if I were to
go this route. If you are giving up on Bash for this that tells me a lot.
In any event, I don't regret having spent time with it as it's always
been on a list of things to mess with. I read your post where you say
you get drawn off to the next thing... I know that feeling well.
> With a 1000-post file,
> it does inserts to the fromtable and headers table, and takes about
> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me.
> The inserts are a real performance bottleneck. How do JS and Python
> do for insert time?
I dunno, I've never done this much data so I've had no need to check.
DFS is the guy to ask for Python. I've heard the current version of Php
is relatively fast at this, Sandman could weigh in on specifics for it.
It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims:
"To my surprise, LOAD DATA INFILE proves faster than a table copy:"
<https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723>
[toc] | [prev] | [next] | [standalone]
| From | owl <owl@rooftop.invalid> |
|---|---|
| Date | 2018-10-09 15:55 +0000 |
| Message-ID | <8z98bbkea98.l97@rooftop.invalid> |
| In reply to | #471293 |
Steve Carroll <fretwizzer@gmail.com> wrote: > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote: >> Steve Carroll <fretwizzer@gmail.com> wrote: >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote: >> >> Steve Carroll <fretwizzer@gmail.com> wrote: >> > >> >> >> It doesn't work right for a large number of posts, and there's a problem >> with the csplit regex, so I'm scrapping the whole shell approach. Too >> much labor to make it not messy, and I don't trust bash arrays for >> large amounts of big data. > > I was thinking of refreshing the array with the processing and writing > of each record but I haven't yet figured out how while keeping things > in sync. I got all the data in an array of structs that have integer member for the article id. > What's the problem with large bash arrays? > They're just messy. And I don't know if they are go on the stack or not. >> I'm working on one in C, with everything in RAM. > > To me that makes more sense (why pound a drive when you have RAM > which is faster anyway?). I was messing with C before I started with this > stuff again but, from what little I saw, the syntax wasn't as grueling as > Bash. I was on step one with C... I didn't get to the point where I was > using system() or anything like that so I'd have to figure out how to get > all these little utility apps (awk, sed, cut, etc.) working in it if I were to > go this route. If you are giving up on Bash for this that tells me a lot. > In any event, I don't regret having spent time with it as it's always > been on a list of things to mess with. I read your post where you say > you get drawn off to the next thing... I know that feeling well. > Not using system(). The code does all parsing and writes. Right now it's just walking the bytes and comparing, but I'm considering using the regex library. Header extraction is easy enough without it though. >> With a 1000-post file, >> it does inserts to the fromtable and headers table, and takes about >> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me. >> The inserts are a real performance bottleneck. How do JS and Python >> do for insert time? > > I dunno, I've never done this much data so I've had no need to check. > DFS is the guy to ask for Python. I've heard the current version of Php > is relatively fast at this, Sandman could weigh in on specifics for it. > > It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims: > "To my surprise, LOAD DATA INFILE proves faster than a table copy:" > > <https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723> Holy shit! I wrapped the inserts in a transaction and now it does 2000 inserts in half a second. And that's big inserts -- full headers in each record for 1000 of them. I think DFS gave me that pointer once before, and I forgot it. anon@lowtide:~/usenet/database$ ./dropdb anon@lowtide:~/usenet/database$ ./createdb anon@lowtide:~/usenet/database$ time ./blah real 0m0.482s user 0m0.168s sys 0m0.080s anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb mysql> select count(*) from headers; count(*) 998 mysql> select count(*) from fromtable; count(*) 998 mysql> select * from fromtable limit 5; id content 2364171 Steve Carroll <frelwizzen@gmail.com>\n 2364172 RonB <ronb02NOSPAM@gmail.com>\n 2364173 Steve Carroll <frelwizzen@gmail.com>\n 2364174 Steve Carroll <frelwizzen@gmail.com>\n 2364175 benj <benj.fake@gmail.com>\n mysql> quit anon@lowtide:~/usenet/database$ I need to strip those trailing newlines out somewhere in the code.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <frelwizzen@gmail.com> |
|---|---|
| Date | 2018-10-09 09:20 -0700 |
| Message-ID | <b5804218-5590-442a-8a00-46f503cc1cbf@googlegroups.com> |
| In reply to | #471296 |
On Tuesday, October 9, 2018 at 8:55:11 AM UTC-7, owl wrote: > Steve Carroll <fretwizzer@gmail.com> wrote: > > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote: > >> Steve Carroll <fretwizzer@gmail.com> wrote: > >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote: > >> >> Steve Carroll <fretwizzer@gmail.com> wrote: > >> > > > >> > >> > >> It doesn't work right for a large number of posts, and there's a problem > >> with the csplit regex, so I'm scrapping the whole shell approach. Too > >> much labor to make it not messy, and I don't trust bash arrays for > >> large amounts of big data. > > > > I was thinking of refreshing the array with the processing and writing > > of each record but I haven't yet figured out how while keeping things > > in sync. > > > I got all the data in an array of structs that have integer member > for the article id. > > > > What's the problem with large bash arrays? > > > > They're just messy. And I don't know if they are go on the stack > or not. > > > >> I'm working on one in C, with everything in RAM. > > > > To me that makes more sense (why pound a drive when you have RAM > > which is faster anyway?). I was messing with C before I started with this > > stuff again but, from what little I saw, the syntax wasn't as grueling as > > Bash. I was on step one with C... I didn't get to the point where I was > > using system() or anything like that so I'd have to figure out how to get > > all these little utility apps (awk, sed, cut, etc.) working in it if I were to > > go this route. If you are giving up on Bash for this that tells me a lot. > > In any event, I don't regret having spent time with it as it's always > > been on a list of things to mess with. I read your post where you say > > you get drawn off to the next thing... I know that feeling well. > > > > Not using system(). The code does all parsing and writes. > Right now it's just walking the bytes and comparing, but > I'm considering using the regex library. Header extraction > is easy enough without it though. > > > >> With a 1000-post file, > >> it does inserts to the fromtable and headers table, and takes about > >> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me. > >> The inserts are a real performance bottleneck. How do JS and Python > >> do for insert time? > > > > I dunno, I've never done this much data so I've had no need to check. > > DFS is the guy to ask for Python. I've heard the current version of Php > > is relatively fast at this, Sandman could weigh in on specifics for it. > > > > It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims: > > "To my surprise, LOAD DATA INFILE proves faster than a table copy:" > > > > <https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723> > > Holy shit! I wrapped the inserts in a transaction and now it does 2000 > inserts in half a second. And that's big inserts -- full headers in each > record for 1000 of them. I think DFS gave me that pointer once before, > and I forgot it. > > anon@lowtide:~/usenet/database$ ./dropdb > anon@lowtide:~/usenet/database$ ./createdb > anon@lowtide:~/usenet/database$ time ./blah > > real 0m0.482s > user 0m0.168s > sys 0m0.080s > anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb > mysql> select count(*) from headers; > count(*) > 998 > mysql> select count(*) from fromtable; > count(*) > 998 > mysql> select * from fromtable limit 5; > id content > 2364171 Steve Carroll <frelwizzen@gmail.com>\n > 2364172 RonB <ronb02NOSPAM@gmail.com>\n > 2364173 Steve Carroll <frelwizzen@gmail.com>\n > 2364174 Steve Carroll <frelwizzen@gmail.com>\n > 2364175 benj <benj.fake@gmail.com>\n > mysql> quit > anon@lowtide:~/usenet/database$ > > I need to strip those trailing newlines out somewhere in the code. Was that meant to be clever? It should not matter because the name on a post does not matter, we know it is from the script-troller anyway. Protected code is best done with code morphing. You guys can only speculate from the perspective of a psychopath. Sandman is trying "really hard" to project their modus operandi onto Klaus Schadenfreude. For some time Sandman has pushed the belief that Klaus Schadenfreude needs 'evidence' to point out all his lies. The fact is that no one needs any reason to do that. So Sandman pulls this insane insulting foolishness in a weak bid to 'boost' the idea that Klaus Schadenfreude is like him. Don't look now, but I think Sandman has a serious obsession with Klaus Schadenfreude. -- Do not click this link! http://www.5z8.info/trojan_p4g1zy_how-to-skin-a-gerbil Jonas Eklundh Communication
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-09 09:39 -0700 |
| Message-ID | <ddb7e1e2-7406-4872-b7b0-0365b2599e1d@googlegroups.com> |
| In reply to | #471296 |
On Tuesday, October 9, 2018 at 9:55:11 AM UTC-6, owl wrote: > Steve Carroll <fretwizzer@gmail.com> wrote: > > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote: > >> Steve Carroll <fretwizzer@gmail.com> wrote: > >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote: > >> >> Steve Carroll <fretwizzer@gmail.com> wrote: > >> > > > >> > >> > >> It doesn't work right for a large number of posts, and there's a problem > >> with the csplit regex, so I'm scrapping the whole shell approach. Too > >> much labor to make it not messy, and I don't trust bash arrays for > >> large amounts of big data. > > > > I was thinking of refreshing the array with the processing and writing > > of each record but I haven't yet figured out how while keeping things > > in sync. > > > I got all the data in an array of structs that have integer member > for the article id. Even the bodies? How'd you determine how to size the structs for those (don't structs need to be sized up front)? > > What's the problem with large bash arrays? > > > > They're just messy. And I don't know if they are go on the stack > or not. > > > >> I'm working on one in C, with everything in RAM. > > > > To me that makes more sense (why pound a drive when you have RAM > > which is faster anyway?). I was messing with C before I started with this > > stuff again but, from what little I saw, the syntax wasn't as grueling as > > Bash. I was on step one with C... I didn't get to the point where I was > > using system() or anything like that so I'd have to figure out how to get > > all these little utility apps (awk, sed, cut, etc.) working in it if I were to > > go this route. If you are giving up on Bash for this that tells me a lot. > > In any event, I don't regret having spent time with it as it's always > > been on a list of things to mess with. I read your post where you say > > you get drawn off to the next thing... I know that feeling well. > > > > Not using system(). The code does all parsing and writes. > Right now it's just walking the bytes and comparing, but > I'm considering using the regex library. Header extraction > is easy enough without it though. Are you comparing to the header labels and the newline char? What about References: and the body content? Many of the posts have a line that begins: On... (some date, someone) wrote: But there are posts that don't have it or anything you could use. I was thinking about blanks lines after the final header, which all seem to use angle brackets (but I could be wrong about that). > >> With a 1000-post file, > >> it does inserts to the fromtable and headers table, and takes about > >> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me. > >> The inserts are a real performance bottleneck. How do JS and Python > >> do for insert time? > > > > I dunno, I've never done this much data so I've had no need to check. > > DFS is the guy to ask for Python. I've heard the current version of Php > > is relatively fast at this, Sandman could weigh in on specifics for it. > > > > It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims: > > "To my surprise, LOAD DATA INFILE proves faster than a table copy:" > > > > <https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723> > > Holy shit! I wrapped the inserts in a transaction and now it does 2000 > inserts in half a second. From 22/sec to 1000/sec increase, not too shabby. > And that's big inserts -- full headers in each > record for 1000 of them. To multiple tables? > I think DFS gave me that pointer once before, > and I forgot it. Getting them as fast as you can via a shell script and writing them like this is definitely a solid way to go for speed. It's sorta looking like you could get a year's worth of posts, process 'em and write it all to a DB in under a minute. > anon@lowtide:~/usenet/database$ ./dropdb > anon@lowtide:~/usenet/database$ ./createdb > anon@lowtide:~/usenet/database$ time ./blah > > real 0m0.482s > user 0m0.168s > sys 0m0.080s > anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb > mysql> select count(*) from headers; > count(*) > 998 > mysql> select count(*) from fromtable; > count(*) > 998 > mysql> select * from fromtable limit 5; > id content > 2364171 Steve Carroll <frelwizzen@gmail.com>\n > 2364172 RonB <ronb02NOSPAM@gmail.com>\n > 2364173 Steve Carroll <frelwizzen@gmail.com>\n > 2364174 Steve Carroll <frelwizzen@gmail.com>\n > 2364175 benj <benj.fake@gmail.com>\n > mysql> quit > anon@lowtide:~/usenet/database$ > > I need to strip those trailing newlines out somewhere in the code.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <fretwizzer@gmail.com> |
|---|---|
| Date | 2018-10-09 09:41 -0700 |
| Message-ID | <b72d21f6-e8cd-45c1-b54e-520866ae4b95@googlegroups.com> |
| In reply to | #471300 |
On Tuesday, October 9, 2018 at 10:39:36 AM UTC-6, Steve Carroll wrote: > On Tuesday, October 9, 2018 at 9:55:11 AM UTC-6, owl wrote: > > Steve Carroll <fretwizzer@gmail.com> wrote: > > > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote: > > >> Steve Carroll <fretwizzer@gmail.com> wrote: > > >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote: > > >> >> Steve Carroll <fretwizzer@gmail.com> wrote: > > >> > > > > > >> > > >> > > >> It doesn't work right for a large number of posts, and there's a problem > > >> with the csplit regex, so I'm scrapping the whole shell approach. Too > > >> much labor to make it not messy, and I don't trust bash arrays for > > >> large amounts of big data. > > > > > > I was thinking of refreshing the array with the processing and writing > > > of each record but I haven't yet figured out how while keeping things > > > in sync. > > > > > > I got all the data in an array of structs that have integer member > > for the article id. > > Even the bodies? How'd you determine how to size the structs for those > (don't structs need to be sized up front)? > > > > What's the problem with large bash arrays? > > > > > > > They're just messy. And I don't know if they are go on the stack > > or not. > > > > > > >> I'm working on one in C, with everything in RAM. > > > > > > To me that makes more sense (why pound a drive when you have RAM > > > which is faster anyway?). I was messing with C before I started with this > > > stuff again but, from what little I saw, the syntax wasn't as grueling as > > > Bash. I was on step one with C... I didn't get to the point where I was > > > using system() or anything like that so I'd have to figure out how to get > > > all these little utility apps (awk, sed, cut, etc.) working in it if I were to > > > go this route. If you are giving up on Bash for this that tells me a lot. > > > In any event, I don't regret having spent time with it as it's always > > > been on a list of things to mess with. I read your post where you say > > > you get drawn off to the next thing... I know that feeling well. > > > > > > > Not using system(). The code does all parsing and writes. > > Right now it's just walking the bytes and comparing, but > > I'm considering using the regex library. Header extraction > > is easy enough without it though. > > Are you comparing to the header labels and the newline char? What > about References: and the body content? Many of the posts have a > line that begins: On... (some date, someone) wrote: > > But there are posts that don't have it or anything you could use. I was > thinking about blanks lines after the final header, which all seem to use > angle brackets (but I could be wrong about that). Edit: I meant a colon, not brackets.
[toc] | [prev] | [next] | [standalone]
| From | Steve Carroll <frelwizzen@gmail.com> |
|---|---|
| Date | 2018-10-09 10:23 -0700 |
| Message-ID | <7abd5b01-a447-4479-9632-604b1c5445bf@googlegroups.com> |
| In reply to | #471300 |
On Tuesday, October 9, 2018 at 9:39:36 AM UTC-7, Steve Carroll wrote: > On Tuesday, October 9, 2018 at 9:55:11 AM UTC-6, owl wrote: > > Steve Carroll <fretwizzer@gmail.com> wrote: > > > On Tuesday, October 9, 2018 at 7:45:21 AM UTC-6, owl wrote: > > >> Steve Carroll <fretwizzer@gmail.com> wrote: > > >> > On Monday, October 8, 2018 at 3:27:41 PM UTC-6, owl wrote: > > >> >> Steve Carroll <fretwizzer@gmail.com> wrote: > > >> > > > > > >> > > >> > > >> It doesn't work right for a large number of posts, and there's a problem > > >> with the csplit regex, so I'm scrapping the whole shell approach. Too > > >> much labor to make it not messy, and I don't trust bash arrays for > > >> large amounts of big data. > > > > > > I was thinking of refreshing the array with the processing and writing > > > of each record but I haven't yet figured out how while keeping things > > > in sync. > > > > > > I got all the data in an array of structs that have integer member > > for the article id. > > Even the bodies? How'd you determine how to size the structs for those > (don't structs need to be sized up front)? > > > > What's the problem with large bash arrays? > > > > > > > They're just messy. And I don't know if they are go on the stack > > or not. > > > > > > >> I'm working on one in C, with everything in RAM. > > > > > > To me that makes more sense (why pound a drive when you have RAM > > > which is faster anyway?). I was messing with C before I started with this > > > stuff again but, from what little I saw, the syntax wasn't as grueling as > > > Bash. I was on step one with C... I didn't get to the point where I was > > > using system() or anything like that so I'd have to figure out how to get > > > all these little utility apps (awk, sed, cut, etc.) working in it if I were to > > > go this route. If you are giving up on Bash for this that tells me a lot. > > > In any event, I don't regret having spent time with it as it's always > > > been on a list of things to mess with. I read your post where you say > > > you get drawn off to the next thing... I know that feeling well. > > > > > > > Not using system(). The code does all parsing and writes. > > Right now it's just walking the bytes and comparing, but > > I'm considering using the regex library. Header extraction > > is easy enough without it though. > > Are you comparing to the header labels and the newline char? What > about References: and the body content? Many of the posts have a > line that begins: On... (some date, someone) wrote: > > But there are posts that don't have it or anything you could use. I was > thinking about blanks lines after the final header, which all seem to use > angle brackets (but I could be wrong about that). > > > >> With a 1000-post file, > > >> it does inserts to the fromtable and headers table, and takes about > > >> 1.5 minutes for the 2000 inserts. That doesn't seem very good to me. > > >> The inserts are a real performance bottleneck. How do JS and Python > > >> do for insert time? > > > > > > I dunno, I've never done this much data so I've had no need to check. > > > DFS is the guy to ask for Python. I've heard the current version of Php > > > is relatively fast at this, Sandman could weigh in on specifics for it. > > > > > > It seems MySQL's 'LOAD DATA INFILE' is the way, this guy claims: > > > "To my surprise, LOAD DATA INFILE proves faster than a table copy:" > > > > > > <https://medium.com/@benmorel/high-speed-inserts-with-mysql-9d3dcd76f723> > > > > Holy shit! I wrapped the inserts in a transaction and now it does 2000 > > inserts in half a second. > > From 22/sec to 1000/sec increase, not too shabby. > > > And that's big inserts -- full headers in each > > record for 1000 of them. > > To multiple tables? > > > I think DFS gave me that pointer once before, > > and I forgot it. > > Getting them as fast as you can via a shell script and writing them > like this is definitely a solid way to go for speed. It's sorta looking > like you could get a year's worth of posts, process 'em and write it > all to a DB in under a minute. > > > anon@lowtide:~/usenet/database$ ./dropdb > > anon@lowtide:~/usenet/database$ ./createdb > > anon@lowtide:~/usenet/database$ time ./blah > > > > real 0m0.482s > > user 0m0.168s > > sys 0m0.080s > > anon@lowtide:~/usenet/database$ mysql -pblah --silent usenetdb > > mysql> select count(*) from headers; > > count(*) > > 998 > > mysql> select count(*) from fromtable; > > count(*) > > 998 > > mysql> select * from fromtable limit 5; > > id content > > 2364171 Steve Carroll <frelwizzen@gmail.com>\n > > 2364172 RonB <ronb02NOSPAM@gmail.com>\n > > 2364173 Steve Carroll <frelwizzen@gmail.com>\n > > 2364174 Steve Carroll <frelwizzen@gmail.com>\n > > 2364175 benj <benj.fake@gmail.com>\n > > mysql> quit > > anon@lowtide:~/usenet/database$ > > > > I need to strip those trailing newlines out somewhere in the code. This is just a pastime of mine. I won't ask Steve Carroll how any part of Klaus Schadenfreude's rear tastes no matter how often Steve Carroll kisses it. Yup. Sadly this is what we have to put up with. Cretins who have no reason for being here other than to flood. Until or unless Steve Carroll offers up his 'top-of-the-line' Linux system for assessment, there is no challenge, just insane contentions. The whole buffonery is chock-full of ridiculous cracks and counts heavily on Steve Carroll's 'liberal education' shenanigans. Sheer BS by an ignorant, misleading, conniving, conspiring peabrain who couldn't tell the truth no matter what his socks/shills say. The Mac has more automation options. -- Puppy Videos https://www.reddit.com/r/linux/comments/6sfkup/what_desktop_tasks_does_linux_handle_better_than/= http://www.5z8.info/hack-outlook_c6c9in_michaelangelo-virus Jonas Eklundh Communication
[toc] | [prev] | [next] | [standalone]
Page 5 of 8 — ← Prev page 1 2 3 4 [5] 6 7 8 Next page →
Back to top | Article view | comp.os.linux.advocacy
csiph-web