Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.advocacy > #349494

Re: Algorithm to find data range

From owl <owl@rooftop.invalid>
Newsgroups comp.os.linux.advocacy
Subject Re: Algorithm to find data range
Date 2016-04-12 10:44 +0000
Organization O.W.L.
Message-ID <hjgi30ara.kgi4r3@rooftop.invalid> (permalink)
References <negfcb$4ee$1@dont-email.me> <sandman-2ada1150bc27e3d11b868223a447a75e@individual.net> <nehpge$pok$1@dont-email.me>

Show all headers | View raw

DFS <nospam@dfs.com> wrote:
> On 4/11/2016 1:09 PM, Sandman wrote:
>>
...
>> In the end, traversing 500k lines of data is just as quick with todays CPU's.
> 
> Sure, but where's the fun in that?  I could download it once, put it in 
> a db table and query it right away.  Even a Linux advocate could do that.
> 
> But downloading and storing 500K rows in a table or in memory is 
> unacceptable if I want a portable system.
> 
> What if I can write code that gets the answers quickly, doesn't require 
> a db, and allows a user to just type:
> 
> -stats sci.math 30 days ending 20080930
> -stats comp.os.linux.advocacy 7 days beginning 20160101
> 
> 
> That's the ticket.
> 

Like this?

anon@lowtide:~$ ./blah.sh localhost cola 20150615 +6days
Earliest date searched:  15 Jun 2015
Latest date searched  :  21 Jun 2015
First ID in range: 2183608 Mon, 15 Jun 2015 00:43:10 +0200
Last ID in range : 2185782 Sun, 21 Jun 2015 22:42:47 -0700
anon@lowtide:~$

anon@lowtide:~$ ./blah.sh localhost cola 20150615 -2months
Earliest date searched:  15 Apr 2015
Latest date searched  :  15 Jun 2015
First ID in range: 2173942 Wed, 15 Apr 2015 01:13:48 +0200
Last ID in range : 2183869 Mon, 15 Jun 2015 21:36:04 -0700 (PDT)
anon@lowtide:~$

anon@lowtide:~$ ./blah.sh localhost alt.test 20151231 -1year
Earliest date searched:  31 Dec 2014
Latest date searched  :  31 Dec 2015
First ID in range: 4645896 Wed, 31 Dec 2014 00:08:41 +0000 (UTC)
Last ID in range : 4747061 Thu, 31 Dec 2015 18:40:06 -0500
anon@lowtide:~$

I'll post the code later after I add some logic to handle leading
zeros on the day portion of the date string.  I have them stripped
in this version and add a leading space to the date query so as not
to have "1 Dec" find  "11 Dec", "21 Dec", etc.  Unfortunately some
date headers have the leading zero and some don't, so with this version
it can end up with a null result on either end.

It's slow.  Takes about 20 some odd seconds to complete.  It will
be slower still with the leading zero code.

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 11:23 -0400
  Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-11 17:09 +0000
    Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 19:25 +0000
      Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-11 20:58 +0000
        Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 17:38 -0400
          Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 18:11 -0400
            Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 22:25 +0000
          Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 06:16 +0000
        Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 21:46 +0000
          Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 07:18 +0000
            Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 08:15 +0000
              Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 10:34 +0000
            Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:32 -0400
              Re: Algorithm to find data range Steve Carroll <fretwizzer@gmail.com> - 2016-04-12 10:39 -0700
                Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:58 -0400
                Re: Algorithm to find data range Steve Carroll <fretwizzer@gmail.com> - 2016-04-12 11:59 -0700
    Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 23:23 -0400
      Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 07:14 +0000
      Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 10:44 +0000
        Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 15:17 +0000
          Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-13 17:15 -0400
            Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-13 22:23 +0000
              Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-13 18:32 -0400
                Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-14 04:33 +0000
                Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-14 05:51 +0000
                Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-14 20:15 -0400
                Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-15 01:18 +0000
        Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:25 -0400
          Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 19:08 +0000
    Re: Algorithm to find data range vallor <vallor@cultnix.org> - 2016-04-12 04:40 +0000
  Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 18:06 +0000
  Re: Algorithm to find data range 7 <7@enemygadgets.com> - 2016-04-11 22:46 +0000
    Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-11 18:52 -0400
      Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:28 -0400
        Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-12 13:46 -0400
    Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 22:13 -0400
  Re: Algorithm to find data range Fabian Russell <fb@zen.info> - 2016-04-11 23:22 +0000
    Re: Algorithm to find data range vallor <vallor@cultnix.org> - 2016-04-12 00:06 +0000
      Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-11 20:26 -0400
      Re: Algorithm to find data range Chris Ahlstrom <OFeem1987@teleworm.us> - 2016-04-12 05:30 -0400
      Re: Algorithm to find data range chrisv <chrisv@nospam.invalid> - 2016-04-12 06:50 -0500
    Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 22:34 -0400
      Re: Algorithm to find data range Fabian Russell <fb@zen.info> - 2016-04-12 09:25 +0000
        Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:32 -0400
  Is it your own personal NNTP/Usenet server ? Jeff-Relf.Me <@.> - 2016-04-11 16:32 -0700
    Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 12:47 -0400
      Re: Algorithm to find data range Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-04-12 20:46 +0200
        Re: Algorithm to find data range chrisv <chrisv@nospam.invalid> - 2016-04-12 13:59 -0500
        Re: Algorithm to find data range Silver Slimer <linux@sucks.balls> - 2016-04-12 17:08 -0400
      My "newsReader" (X.ZIP) is also a console. Jeff-Relf.Me <@.> - 2016-04-12 12:27 -0700
      My "newsReader" (X.ZIP) is also a console. Jeff-Relf.Me <@.> - 2016-04-12 12:31 -0700

csiph-web