Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #46262

Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines()

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

On 28 May 2013 04:19, "Bryan Britten" <britten.bryan@gmail.com> wrote:
> I'm not familiar with using read(4096), I'll have to look into that. When
I tried to just save the file, my computer just sat in limbo for some time
and didn't seem to want to process the command.

That's just file.read with an integer argument. You can read a file by
chunks by repeatedly calling that function until you get the empty string.

> Based on my *extremely* limited knowledge of JSON, that's definitely the
type of file this is. Here is a snippet of what is seen when you log in:
...
That's json. It's pretty big, but not big enough to stall a slow computer
more than half a second.

-

I've looked for documentation on that method on twitter.

It seems that it's part of the twitter streaming api.

https://dev.twitter.com/docs/streaming-apis

What this means is that the requests aren't supposed to end. They are
supposed to be read gradually, using the lines to split the response into
meaningful chunks. That's why you can't read the data and why your browser
never gets around to download it. Both urlopen and your browser block while
waiting for the request to end.

Here's more info on streaming requests on their docs:

https://dev.twitter.com/docs/streaming-apis/processing

For streaming requests in python, I would point you to the requests
library, but I am not sure it handles streaming requests.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Bryan Britten <britten.bryan@gmail.com> - 2013-05-27 13:47 -0700
  Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Roy Smith <roy@panix.com> - 2013-05-27 16:56 -0400
    Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Bryan Britten <britten.bryan@gmail.com> - 2013-05-27 14:29 -0700
      Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Denis McMahon <denismfmcmahon@gmail.com> - 2013-05-27 21:35 +0000
      Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Fábio Santos <fabiosantosart@gmail.com> - 2013-05-28 00:36 +0100
  Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Dave Angel <davea@davea.name> - 2013-05-27 19:58 -0400
    Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Bryan Britten <britten.bryan@gmail.com> - 2013-05-27 20:11 -0700
      Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Fábio Santos <fabiosantosart@gmail.com> - 2013-05-28 08:31 +0100
        Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Bryan Britten <britten.bryan@gmail.com> - 2013-05-28 07:32 -0700
        Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Alister <alister.ware@ntlworld.com> - 2013-05-28 17:52 +0000
  Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-05-27 21:40 -0400

csiph-web