Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'api.': 0.05; 'string.': 0.05; 'meaningful': 0.09; 'second.': 0.09; 'snippet': 0.09; 'subject:()': 0.09; 'cc:addr:python-list': 0.11; 'argument.': 0.16; 'big,': 0.16; 'chunks': 0.16; 'chunks.': 0.16; 'command.': 0.16; 'json,': 0.16; 'subject:Reading': 0.16; 'subject:URL': 0.16; 'subject:versus': 0.16; 'twitter.': 0.16; 'wrote:': 0.18; 'looked': 0.18; 'file,': 0.19; 'split': 0.19; 'seems': 0.21; 'email addr:gmail.com>': 0.22; 'handles': 0.22; 'cc:addr:python.org': 0.22; "aren't": 0.24; 'integer': 0.24; 'library,': 0.24; 'url:dev': 0.24; 'cc:2**0': 0.24; 'cc:no real name:2**0': 0.24; "i've": 0.25; '>': 0.26; 'gets': 0.27; 'header:In-Reply-To:1': 0.27; 'tried': 0.27; 'point': 0.28; 'function': 0.29; 'streaming': 0.30; 'message-id:@mail.gmail.com': 0.30; "i'm": 0.30; 'lines': 0.31; 'requests': 0.31; 'that.': 0.31; 'file': 0.32; 'supposed': 0.32; 'subject:from': 0.34; 'knowledge': 0.35; "can't": 0.35; 'info': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'in:': 0.36; 'sat': 0.36; "didn't": 0.36; 'method': 0.36; "i'll": 0.36; 'half': 0.37; 'sure': 0.39; 'enough': 0.39; 'read': 0.60; 'is.': 0.60; 'browser': 0.61; 'save': 0.62; 'more': 0.64; 'to:addr:gmail.com': 0.65; 'here': 0.66; 'end.': 0.84; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=L0dPk3jxmo4C5+sUBrzGoZnXl3Vx5NQUGdZr9m+4Pco=; b=Iw4s79gc6luij+cbDrKvapIzAizfv99fpJ2N0b6RcO5jD0FYNrSM/pfSHlRHjCfEzG oZqnE9JI6bQ1gM22xqCRF8o22mduFWgne4bxrOwdndoEODKZh39wnVPi3Xi15zL+oITA vHj04VOGxpUrd21yyTB8t6kSGMkHKig+NjAJDK4n5Y79rNVpj1vFt4Y3CihYv1lr3CMC NNQjieb+ps09P1UxPx4IU0Raku9YRN/7ah6RmzhS2PACfz1+27yzWhqb0Ey0curI47Ix Ng3G8K4CI8fWjGD/JgRKfZAVcJJ99STCWe7qcMVXZdRQ2Ed3/cWzp85wFWpB89nAT18J xHSA== MIME-Version: 1.0 X-Received: by 10.49.107.2 with SMTP id gy2mr35815878qeb.21.1369726295388; Tue, 28 May 2013 00:31:35 -0700 (PDT) In-Reply-To: <4db5a3be-d9dc-455c-8e3b-5adebad2dcdd@googlegroups.com> References: <10be5c62-4c58-4b4f-b00a-82d85ee4ef8e@googlegroups.com> <4db5a3be-d9dc-455c-8e3b-5adebad2dcdd@googlegroups.com> Date: Tue, 28 May 2013 08:31:35 +0100 Subject: Re: Reading *.json from URL - json.loads() versus urllib.urlopen.readlines() From: =?ISO-8859-1?Q?F=E1bio_Santos?= To: Bryan Britten Content-Type: multipart/alternative; boundary=047d7bdc13ce46d1f204ddc241a8 Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 78 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1369726304 news.xs4all.nl 16012 [2001:888:2000:d::a6]:59358 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:46262 --047d7bdc13ce46d1f204ddc241a8 Content-Type: text/plain; charset=ISO-8859-1 On 28 May 2013 04:19, "Bryan Britten" wrote: > I'm not familiar with using read(4096), I'll have to look into that. When I tried to just save the file, my computer just sat in limbo for some time and didn't seem to want to process the command. That's just file.read with an integer argument. You can read a file by chunks by repeatedly calling that function until you get the empty string. > Based on my *extremely* limited knowledge of JSON, that's definitely the type of file this is. Here is a snippet of what is seen when you log in: ... That's json. It's pretty big, but not big enough to stall a slow computer more than half a second. - I've looked for documentation on that method on twitter. It seems that it's part of the twitter streaming api. https://dev.twitter.com/docs/streaming-apis What this means is that the requests aren't supposed to end. They are supposed to be read gradually, using the lines to split the response into meaningful chunks. That's why you can't read the data and why your browser never gets around to download it. Both urlopen and your browser block while waiting for the request to end. Here's more info on streaming requests on their docs: https://dev.twitter.com/docs/streaming-apis/processing For streaming requests in python, I would point you to the requests library, but I am not sure it handles streaming requests. --047d7bdc13ce46d1f204ddc241a8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable


On 28 May 2013 04:19, "Bryan Britten" <britten.bryan@gmail.com> wrote:
> I'm not familiar with using read(4096), I'll have to look into= that. When I tried to just save the file, my computer just sat in limbo fo= r some time and didn't seem to want to process the command.

That's just file.read with an integer argument. You can = read a file by chunks by repeatedly calling that function until you get the= empty string.

> Based on my *extremely* limited knowledge of JSON, that= 's definitely the type of file this is. Here is a snippet of what is se= en when you log in:
...
That's json. It's pretty big, but not big enough to stall a slow co= mputer more than half a second.

-

I've looked for documentation on that method on twitter.=

It seems that it's part of the twitter streaming api.

http= s://dev.twitter.com/docs/streaming-apis

What this means is that the requests aren't supposed to = end. They are supposed to be read gradually, using the lines to split the r= esponse into meaningful chunks. That's why you can't read the data = and why your browser never gets around to download it. Both urlopen and you= r browser block while waiting for the request to end.

Here's more info on streaming requests on their docs:

https://dev.twitter.com/docs/streaming-apis/processing

For streaming requests in python, I would point you to the r= equests library, but I am not sure it handles streaming requests.

--047d7bdc13ce46d1f204ddc241a8--