Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.advocacy > #349582
| From | owl <owl@rooftop.invalid> |
|---|---|
| Newsgroups | comp.os.linux.advocacy |
| Subject | Re: Algorithm to find data range |
| Date | 2016-04-12 19:08 +0000 |
| Organization | O.W.L. |
| Message-ID | <ghjdi03a.f3@rooftop.invalid> (permalink) |
| References | <negfcb$4ee$1@dont-email.me> <sandman-2ada1150bc27e3d11b868223a447a75e@individual.net> <nehpge$pok$1@dont-email.me> <hjgi30ara.kgi4r3@rooftop.invalid> <nejarm$p7n$2@dont-email.me> |
DFS <nospam@dfs.com> wrote: > On 4/12/2016 6:44 AM, owl wrote: >> DFS <nospam@dfs.com> wrote: >>> On 4/11/2016 1:09 PM, Sandman wrote: >>>> >> ... >>>> In the end, traversing 500k lines of data is just as quick with todays CPU's. >>> >>> Sure, but where's the fun in that? I could download it once, put it in >>> a db table and query it right away. Even a Linux advocate could do that. >>> >>> But downloading and storing 500K rows in a table or in memory is >>> unacceptable if I want a portable system. >>> >>> What if I can write code that gets the answers quickly, doesn't require >>> a db, and allows a user to just type: >>> >>> -stats sci.math 30 days ending 20080930 >>> -stats comp.os.linux.advocacy 7 days beginning 20160101 >>> >>> >>> That's the ticket. >>> >> >> Like this? > > Much much much more than that. It looks like you're just finding the > IDs corresponding to the date range entered (which is what this thread > is about). When I/we find a good algorithm, I expect that part to take > no more than 3 seconds no matter how many days are covered. > > My -stats will be a version of the weekly stats we've seen here for > years. I'll release the code and anyone can use it to summarize any > group over any time period (where any is some arbitrary limit you choose > - typically up to one month. You don't want to hog server cycles. And > apparently some servers limit your data downloads.) > > >> anon@lowtide:~$ ./blah.sh localhost cola 20150615 +6days >> Earliest date searched: 15 Jun 2015 >> Latest date searched : 21 Jun 2015 >> First ID in range: 2183608 Mon, 15 Jun 2015 00:43:10 +0200 >> Last ID in range : 2185782 Sun, 21 Jun 2015 22:42:47 -0700 >> anon@lowtide:~$ >> >> anon@lowtide:~$ ./blah.sh localhost cola 20150615 -2months >> Earliest date searched: 15 Apr 2015 >> Latest date searched : 15 Jun 2015 >> First ID in range: 2173942 Wed, 15 Apr 2015 01:13:48 +0200 >> Last ID in range : 2183869 Mon, 15 Jun 2015 21:36:04 -0700 (PDT) >> anon@lowtide:~$ >> >> anon@lowtide:~$ ./blah.sh localhost alt.test 20151231 -1year >> Earliest date searched: 31 Dec 2014 >> Latest date searched : 31 Dec 2015 >> First ID in range: 4645896 Wed, 31 Dec 2014 00:08:41 +0000 (UTC) >> Last ID in range : 4747061 Thu, 31 Dec 2015 18:40:06 -0500 >> anon@lowtide:~$ >> >> I'll post the code later after I add some logic to handle leading >> zeros on the day portion of the date string. I have them stripped >> in this version and add a leading space to the date query so as not >> to have "1 Dec" find "11 Dec", "21 Dec", etc. Unfortunately some >> date headers have the leading zero and some don't, so with this version >> it can end up with a null result on either end. >> >> It's slow. Takes about 20 some odd seconds to complete. It will >> be slower still with the leading zero code. > > > 20 seconds seems way slow. It's probably not in your code. xpat? If > xpat is this slow, it won't be the ticket! > Nah, it's `expect` that's slowing it. Didn't realize it has a default 10 sec timeout. I set it to 1 and speed goes from ~45 sec on the offset search to about 9 sec; and from 20 some to about 5 sec on a single day search. I played with the code a bit more, and for the life of me I cannot get it to accept a [^1-3]N pattern in the handoff to an `expect` send. When it's echo'ed it shows exactly as I would type it at the console, but for some reason it wants to treat the pattern as a command. No amount of escaping seems to help. It's unfortunate, because this would cut out the two extra `expect` runs that I have to do to handle the leading zeros issue. > I have a big python module (uses nntplib, xover, xhdr mostly) that does > tons of processing and even when run against 5000 posts it runs in 2-3 > seconds. > From telnet, I get similar. Hell, I could probably manually go through the steps at the console as fast as I can automate it with expect.
Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 11:23 -0400
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-11 17:09 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 19:25 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-11 20:58 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 17:38 -0400
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 18:11 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 22:25 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 06:16 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 21:46 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 07:18 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 08:15 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 10:34 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:32 -0400
Re: Algorithm to find data range Steve Carroll <fretwizzer@gmail.com> - 2016-04-12 10:39 -0700
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:58 -0400
Re: Algorithm to find data range Steve Carroll <fretwizzer@gmail.com> - 2016-04-12 11:59 -0700
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 23:23 -0400
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 07:14 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 10:44 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 15:17 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-13 17:15 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-13 22:23 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-13 18:32 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-14 04:33 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-14 05:51 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-14 20:15 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-15 01:18 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:25 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 19:08 +0000
Re: Algorithm to find data range vallor <vallor@cultnix.org> - 2016-04-12 04:40 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 18:06 +0000
Re: Algorithm to find data range 7 <7@enemygadgets.com> - 2016-04-11 22:46 +0000
Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-11 18:52 -0400
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:28 -0400
Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-12 13:46 -0400
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 22:13 -0400
Re: Algorithm to find data range Fabian Russell <fb@zen.info> - 2016-04-11 23:22 +0000
Re: Algorithm to find data range vallor <vallor@cultnix.org> - 2016-04-12 00:06 +0000
Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-11 20:26 -0400
Re: Algorithm to find data range Chris Ahlstrom <OFeem1987@teleworm.us> - 2016-04-12 05:30 -0400
Re: Algorithm to find data range chrisv <chrisv@nospam.invalid> - 2016-04-12 06:50 -0500
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 22:34 -0400
Re: Algorithm to find data range Fabian Russell <fb@zen.info> - 2016-04-12 09:25 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:32 -0400
Is it your own personal NNTP/Usenet server ? Jeff-Relf.Me <@.> - 2016-04-11 16:32 -0700
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 12:47 -0400
Re: Algorithm to find data range Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-04-12 20:46 +0200
Re: Algorithm to find data range chrisv <chrisv@nospam.invalid> - 2016-04-12 13:59 -0500
Re: Algorithm to find data range Silver Slimer <linux@sucks.balls> - 2016-04-12 17:08 -0400
My "newsReader" (X.ZIP) is also a console. Jeff-Relf.Me <@.> - 2016-04-12 12:27 -0700
My "newsReader" (X.ZIP) is also a console. Jeff-Relf.Me <@.> - 2016-04-12 12:31 -0700
Re: My "newsReader" (X.ZIP) is also a console. Unknown <dog@gmail.com> - 2016-06-14 00:48 +0000
csiph-web