Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.advocacy > #349521
| From | owl <owl@rooftop.invalid> |
|---|---|
| Newsgroups | comp.os.linux.advocacy |
| Subject | Re: Algorithm to find data range |
| Date | 2016-04-12 15:17 +0000 |
| Organization | O.W.L. |
| Message-ID | <fhjg0a9a.aer3@rooftop.invalid> (permalink) |
| References | <negfcb$4ee$1@dont-email.me> <sandman-2ada1150bc27e3d11b868223a447a75e@individual.net> <nehpge$pok$1@dont-email.me> <hjgi30ara.kgi4r3@rooftop.invalid> |
owl <owl@rooftop.invalid> wrote:
> DFS <nospam@dfs.com> wrote:
>> On 4/11/2016 1:09 PM, Sandman wrote:
>>>
> ...
>>> In the end, traversing 500k lines of data is just as quick with todays CPU's.
>>
>> Sure, but where's the fun in that? I could download it once, put it in
>> a db table and query it right away. Even a Linux advocate could do that.
>>
>> But downloading and storing 500K rows in a table or in memory is
>> unacceptable if I want a portable system.
>>
>> What if I can write code that gets the answers quickly, doesn't require
>> a db, and allows a user to just type:
>>
>> -stats sci.math 30 days ending 20080930
>> -stats comp.os.linux.advocacy 7 days beginning 20160101
>>
>>
>> That's the ticket.
>>
>
> Like this?
>
> anon@lowtide:~$ ./blah.sh localhost cola 20150615 +6days
> Earliest date searched: 15 Jun 2015
> Latest date searched : 21 Jun 2015
> First ID in range: 2183608 Mon, 15 Jun 2015 00:43:10 +0200
> Last ID in range : 2185782 Sun, 21 Jun 2015 22:42:47 -0700
> anon@lowtide:~$
>
> anon@lowtide:~$ ./blah.sh localhost cola 20150615 -2months
> Earliest date searched: 15 Apr 2015
> Latest date searched : 15 Jun 2015
> First ID in range: 2173942 Wed, 15 Apr 2015 01:13:48 +0200
> Last ID in range : 2183869 Mon, 15 Jun 2015 21:36:04 -0700 (PDT)
> anon@lowtide:~$
>
> anon@lowtide:~$ ./blah.sh localhost alt.test 20151231 -1year
> Earliest date searched: 31 Dec 2014
> Latest date searched : 31 Dec 2015
> First ID in range: 4645896 Wed, 31 Dec 2014 00:08:41 +0000 (UTC)
> Last ID in range : 4747061 Thu, 31 Dec 2015 18:40:06 -0500
> anon@lowtide:~$
>
> I'll post the code later after I add some logic to handle leading
> zeros on the day portion of the date string. I have them stripped
> in this version and add a leading space to the date query so as not
> to have "1 Dec" find "11 Dec", "21 Dec", etc. Unfortunately some
> date headers have the leading zero and some don't, so with this version
> it can end up with a null result on either end.
>
> It's slow. Takes about 20 some odd seconds to complete. It will
> be slower still with the leading zero code.
>
OK. The code's below. Not doing every conceivable sanity check,
but probably enough unless you're trying to break it.
anon@lowtide:~$ ./blah.sh
usage: ./blah.sh <server> <newsgroup> <YYYYMMDD> [date offset]
anon@lowtide:~$
The optional date offset should be something like +1day, -2weeks
It depends on `expect`, so you may have to install that.
It gets server login credentials from .newsauth file (lines in this form):
serverA password username
serverB password username
serverC password username
Performance is horrible, but it seems to work OK.
You'll think it's hung, but it's not.
Takes anywhere from 20-50 seconds to return.
Oh yeah, almost forgot: Linux FTW. ;)
---------------------------------------------------------
#!/bin/bash
if [ ${#} -lt 3 ]; then
echo "usage: ${0} <server> <newsgroup> <YYYYMMDD> [date offset]"
exit
fi
if [ ${2} = "cola" ];then
GROUP="comp.os.linux.advocacy"
else
GROUP=${2}
fi
SERVER=${1}
LOGIN=$(grep ${1} ~/.newsauth |awk '{print $3}')
PASSWORD=$(grep ${1} ~/.newsauth |awk '{print $2}' |sed -e 's/\$/\\$/g')
END_ONE=$(date -d "${3}" "+ %-d %b %Y")
END_ONE_LEADING=$(date -d "${3}" "+%d %b %Y")
END_ONE_SECS=$(date -d "${3}" "+%s")
END_TWO=$(date -d "${3} ${4}" "+ %-d %b %Y")
END_TWO_LEADING=$(date -d "${3} ${4}" "+%d %b %Y")
END_TWO_SECS=$(date -d "${3} ${4}" "+%s")
if [ ${END_ONE_SECS} -lt ${END_TWO_SECS} ];then
EARLIEST=${END_ONE}
EARLIEST_LEADING=${END_ONE_LEADING}
LATEST=${END_TWO}
LATEST_LEADING=${END_TWO_LEADING}
else
EARLIEST=${END_TWO}
EARLIEST_LEADING=${END_TWO_LEADING}
LATEST=${END_ONE}
LATEST_LEADING=${END_ONE_LEADING}
fi
echo "Earliest date searched: ${EARLIEST}"
echo "Latest date searched : ${LATEST}"
GROUPINFO=$(tempfile)
START_DATE_LIST=$(tempfile)
START_DATE_LIST_LEADING=$(tempfile)
if [ ${#} -ne 4 ];then
END_DATE_LIST=${START_DATE_LIST}
END_DATE_LIST_LEADING=${START_DATE_LIST_LEADING}
else
END_DATE_LIST=$(tempfile)
END_DATE_LIST_LEADING=$(tempfile)
fi
expect -c "
spawn telnet ${SERVER} 119
expect \"200\"
send \"authinfo user ${LOGIN}\r\"
expect \"381 PASS required\"
send \"authinfo pass ${PASSWORD}\r\"
expect \"281 Ok\"
send \"group ${GROUP}\r\"
expect \"211\"
send \"quit\r\"
" > ${GROUPINFO}
LOWER=$(tail -n 1 ${GROUPINFO} | awk '{print $3}')
UPPER=$(tail -n 1 ${GROUPINFO} | awk '{print $4}')
expect -c "
spawn telnet ${SERVER} 119
expect \"200\"
send \"authinfo user ${LOGIN}\r\"
expect \"381 PASS required\"
send \"authinfo pass ${PASSWORD}\r\"
expect \"281 Ok\"
send \"group ${GROUP}\r\"
expect \"211\"
send \"xpat date ${LOWER}-${UPPER} *${EARLIEST}*\r\"
expect \"^205 .\"
send \"quit\r\"
" > ${START_DATE_LIST}
expect -c "
spawn telnet ${SERVER} 119
expect \"200\"
send \"authinfo user ${LOGIN}\r\"
expect \"381 PASS required\"
send \"authinfo pass ${PASSWORD}\r\"
expect \"281 Ok\"
send \"group ${GROUP}\r\"
expect \"211\"
send \"xpat date ${LOWER}-${UPPER} *${EARLIEST_LEADING}*\r\"
expect \"^205 .\"
send \"quit\r\"
" > ${START_DATE_LIST_LEADING}
if [ ${#} -eq 4 ]; then
expect -c "
spawn telnet ${SERVER} 119
expect \"200\"
send \"authinfo user ${LOGIN}\r\"
expect \"381 PASS required\"
send \"authinfo pass ${PASSWORD}\r\"
expect \"281 Ok\"
send \"group ${GROUP}\r\"
expect \"211\"
send \"xpat date ${LOWER}-${UPPER} *${LATEST}*\r\"
expect \"^205 .\"
send \"quit\r\"
" > ${END_DATE_LIST}
expect -c "
spawn telnet ${SERVER} 119
expect \"200\"
send \"authinfo user ${LOGIN}\r\"
expect \"381 PASS required\"
send \"authinfo pass ${PASSWORD}\r\"
expect \"281 Ok\"
send \"group ${GROUP}\r\"
expect \"211\"
send \"xpat date ${LOWER}-${UPPER} *${LATEST_LEADING}*\r\"
expect \"^205 .\"
send \"quit\r\"
" > ${END_DATE_LIST_LEADING}
fi
FIRST_IN_RANGE=$(grep "${EARLIEST}" ${START_DATE_LIST} |grep -v xpat | head -n 1)
FIRST_IN_RANGE_VAL=$(echo ${FIRST_IN_RANGE} | cut -f1 -d' ')
FIRST_IN_RANGE_LEADING=$(grep "${EARLIEST_LEADING}" ${START_DATE_LIST_LEADING} |grep -v xpat | head -n 1)
FIRST_IN_RANGE_LEADING_VAL=$(echo ${FIRST_IN_RANGE_LEADING} | cut -f1 -d' ')
LAST_IN_RANGE=$(grep "${LATEST}" ${END_DATE_LIST} |grep -v xpat |tail -n 1)
LAST_IN_RANGE_VAL=$(echo ${LAST_IN_RANGE} |cut -f1 -d' ')
LAST_IN_RANGE_LEADING=$(grep "${LATEST_LEADING}" ${END_DATE_LIST_LEADING} |grep -v xpat |tail -n 1)
LAST_IN_RANGE_LEADING_VAL=$(echo ${LAST_IN_RANGE_LEADING} |cut -f1 -d' ')
if [ ${FIRST_IN_RANGE_LEADING_VAL} -lt ${FIRST_IN_RANGE_VAL} ]; then
RANGE_BEGIN=${FIRST_IN_RANGE_LEADING}
else
RANGE_BEGIN=${FIRST_IN_RANGE}
fi
if [ ${LAST_IN_RANGE_LEADING_VAL} -gt ${LAST_IN_RANGE_VAL} ];then
RANGE_END=${LAST_IN_RANGE_LEADING}
else
RANGE_END=${LAST_IN_RANGE}
fi
echo "First ID in range: ${RANGE_BEGIN}"
echo "Last ID in range : ${RANGE_END}"
rm $GROUPINFO
rm $START_DATE_LIST
rm $START_DATE_LIST_LEADING
if [ ${#} -eq 4 ];then
rm $END_DATE_LIST
rm $END_DATE_LIST_LEADING
fi
---------------------------------------------------------
Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 11:23 -0400
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-11 17:09 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 19:25 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-11 20:58 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 17:38 -0400
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 18:11 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 22:25 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 06:16 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 21:46 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 07:18 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 08:15 +0000
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 10:34 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:32 -0400
Re: Algorithm to find data range Steve Carroll <fretwizzer@gmail.com> - 2016-04-12 10:39 -0700
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:58 -0400
Re: Algorithm to find data range Steve Carroll <fretwizzer@gmail.com> - 2016-04-12 11:59 -0700
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 23:23 -0400
Re: Algorithm to find data range Sandman <mr@sandman.net> - 2016-04-12 07:14 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 10:44 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 15:17 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-13 17:15 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-13 22:23 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-13 18:32 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-14 04:33 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-14 05:51 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-14 20:15 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-15 01:18 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:25 -0400
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-12 19:08 +0000
Re: Algorithm to find data range vallor <vallor@cultnix.org> - 2016-04-12 04:40 +0000
Re: Algorithm to find data range owl <owl@rooftop.invalid> - 2016-04-11 18:06 +0000
Re: Algorithm to find data range 7 <7@enemygadgets.com> - 2016-04-11 22:46 +0000
Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-11 18:52 -0400
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:28 -0400
Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-12 13:46 -0400
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 22:13 -0400
Re: Algorithm to find data range Fabian Russell <fb@zen.info> - 2016-04-11 23:22 +0000
Re: Algorithm to find data range vallor <vallor@cultnix.org> - 2016-04-12 00:06 +0000
Re: Algorithm to find data range Omar <omarsayeed@linuxmail.org> - 2016-04-11 20:26 -0400
Re: Algorithm to find data range Chris Ahlstrom <OFeem1987@teleworm.us> - 2016-04-12 05:30 -0400
Re: Algorithm to find data range chrisv <chrisv@nospam.invalid> - 2016-04-12 06:50 -0500
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-11 22:34 -0400
Re: Algorithm to find data range Fabian Russell <fb@zen.info> - 2016-04-12 09:25 +0000
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 13:32 -0400
Is it your own personal NNTP/Usenet server ? Jeff-Relf.Me <@.> - 2016-04-11 16:32 -0700
Re: Algorithm to find data range DFS <nospam@dfs.com> - 2016-04-12 12:47 -0400
Re: Algorithm to find data range Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-04-12 20:46 +0200
Re: Algorithm to find data range chrisv <chrisv@nospam.invalid> - 2016-04-12 13:59 -0500
Re: Algorithm to find data range Silver Slimer <linux@sucks.balls> - 2016-04-12 17:08 -0400
My "newsReader" (X.ZIP) is also a console. Jeff-Relf.Me <@.> - 2016-04-12 12:27 -0700
My "newsReader" (X.ZIP) is also a console. Jeff-Relf.Me <@.> - 2016-04-12 12:31 -0700
Re: My "newsReader" (X.ZIP) is also a console. Unknown <dog@gmail.com> - 2016-06-14 00:48 +0000
csiph-web