Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.advocacy > #355694

Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket

Show all headers | View raw

DFS <nospam@dfs.com> wrote:
> This is a tremendous hassle.
> 

Yes it is.

> Just reliably getting back a full set of data (article IDs and dates) to 
> work with is a nightmare.
> 

Yes.

> Has taken me too much time, but I think 'method B' with a buffer size of 
> 16384 works OK, part of the time anyway (I still have some data parsing 
> to do).
> 
> How are you getting back reliable lists of article IDs and dates from 
> the news server?
> 

Not reliably.  For some date ranges, it works perfectly (at least as far
as it not hanging goes).  Other times not.  The count for a date range
is always going to be just an estimate without a shitload of coding
to cover all possible situations.  And if you're doing it in python,
you'll run into python's fucktardedness with respect to timezones.

> I tried the MS Telnet Client, but saw no way to save the results to a file.
> 

I use tee for that.  I believe the gnu coreutils for win32 has it.
(telnet localhost 119 | tee newsout)


> Note: the xpat query in the code below should return 239 IDs:
> 584121 Fri, 13 May 2016 00:19:20 +0200
> to
> 584453 Fri, 13 May 2016 23:53:44 -0700.

I get the same begin and end dates here, but count of 241.

And note that these are not all actually 13 May UTC.
That's why you'll probably need to grab one day before and
one day after.  For a date range, get the day before the
earliest date and the day after the latest date.

> =============================================================
> 
> import socket, time, re
> 
> s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
> bufSize = 16384
>        
> def recvall(sock, count):
>        buf = b''
>        while count:
>                newbuf = sock.recv(count)
>                if not newbuf: return None
>                buf += newbuf + " "
>                count -= len(newbuf)
>                time.sleep(0.1)
>        return buf
>        
> s.connect(("news.eternal-september.org",119))
> conn = s.recv(bufSize)
> s.send("authinfo user DFS\n")
> auth = s.recv(bufSize)
> s.send("authinfo pass noquarter\n")
> pwd = s.recv(bufSize)
> s.send("group comp.os.linux.advocacy\n")
> resp = s.recv(bufSize)
> resp = resp.split()
> firstGroupID = resp[2]
> lastGroupID = resp[3]
> 
> 
> #get articleIDs for the beginning date
> beginDate = "13 May 2016"
> xpatStr = "xpat date %s-%s *%s*\n\n" % (firstGroupID, lastGroupID, 
> beginDate)
> print xpatStr
> 
> #A. receive all into a string
> s.send(xpatStr)
> hdr  = s.recv(bufSize)
> time.sleep(0.1)
> d = recvall(s, bufSize)
> print "A. receive all into a string"
> print d
> print "-------------------------------------------------------------"
> print
> 
> #B. receive chunks into list object
> s.send(xpatStr)
> hdr  = s.recv(bufSize)
> time.sleep(0.1)
> data = []
> data.append(s.recv(bufSize))
> data.append(s.recv(bufSize))
> data.append(s.recv(bufSize))

I put those in a while loop and check for the '.' on a line
by itself (which sometimes is matchable, and sometimes not :), so
I had to put about five different regexes to test for various
conditions.  I don't even know why this is like this, but
running a hex editor on the showed minor differences in what
should have been matching conditions.

> print "B. receive chunks into list"
> print str(data)
> print "-------------------------------------------------------------"
> print
> 
> #C. print to screen
> s.send(xpatStr)
> hdr  = s.recv(bufSize)
> time.sleep(0.1)
> print "C. print to screen"
> print s.recv(bufSize)
> print s.recv(bufSize)
> print s.recv(bufSize)
> print "-------------------------------------------------------------"
> print
> 
> s.close()
> 
> =============================================================

All in all, it's a pain in the ass.  You might just want to stick  
with recent articles instead of shooting for the flexibility
of being able to grab any arbitrary date range.  I know I got
tired of bothering with it.  It is doable, but I just don't want
to put in the effort, at least not with python.

Back to comp.os.linux.advocacy | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Ping owl: how are you dealing with xpat/NNTP data from the stream socket DFS <nospam@dfs.com> - 2016-05-16 18:54 -0400
  Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket owl <owl@rooftop.invalid> - 2016-05-17 04:09 +0000
    Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket Snit <usenet@gallopinginsanity.com> - 2016-05-16 21:55 -0700
      Snit digest 238 / 2016-05-17 Sandman <mr@sandman.net> - 2016-05-17 06:54 +0000
      Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket Peter Köhlmann <peter-koehlmann@t-online.de> - 2016-05-17 09:13 +0200
        Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket Steve Carroll <fretwizzer@gmail.com> - 2016-05-17 08:28 -0700
        Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket Snit <usenet@gallopinginsanity.com> - 2016-05-17 09:32 -0700
          Snit digest 239 / 2016-05-18 Sandman <mr@sandman.net> - 2016-05-18 05:06 +0000
  Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket Sandman <mr@sandman.net> - 2016-05-17 06:10 +0000
    Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket DFS <nospam@dfs.com> - 2016-05-19 09:01 -0400
      Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket Sandman <mr@sandman.net> - 2016-05-19 15:04 +0000
  Re: Ping owl: how are you dealing with xpat/NNTP data from the stream socket owl <owl@rooftop.invalid> - 2016-05-18 00:18 +0000

csiph-web