Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8625 > unrolled thread

urllib, urlretrieve method, how to get headers?

Back to article view | Back to comp.lang.python

Contents

  urllib, urlretrieve method, how to get headers? Даниил Рыжков <daniil.re@gmail.com> - 2011-07-01 18:03 +1100
    Re: urllib, urlretrieve method, how to get headers? Peter Otten <__peter__@web.de> - 2011-07-01 09:43 +0200
      Re: urllib, urlretrieve method, how to get headers? Даниил Рыжков <daniil.re@gmail.com> - 2011-07-01 19:26 +1100
      Re: urllib, urlretrieve method, how to get headers? Даниил Рыжков <daniil.re@gmail.com> - 2011-07-01 19:53 +1100
      Re: urllib, urlretrieve method, how to get headers? Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2011-07-01 22:31 +0530
      Re: urllib, urlretrieve method, how to get headers? Chris Rebert <clp2@rebertia.com> - 2011-07-01 10:02 -0700
      Re: urllib, urlretrieve method, how to get headers? Даниил Рыжков <daniil.re@gmail.com> - 2011-07-02 11:47 +1100

#8625 — urllib, urlretrieve method, how to get headers?

Hello, everyone!

How can I get headers with urlretrieve? I want to send request and get
headers with necessary information before I execute urlretrieve(). Or
are there any alternatives for urlretrieve()?

-- 
Regards,
Daniil

[toc] | [next] | [standalone]

#8629

Даниил Рыжков wrote:

> How can I get headers with urlretrieve? I want to send request and get
> headers with necessary information before I execute urlretrieve(). Or
> are there any alternatives for urlretrieve()?
 
It's easy to do it manually:

>>> import urllib2

Connect to website and inspect headers:

>>> f = urllib2.urlopen("http://www.python.org")
>>> f.headers["Content-Type"]
'text/html'

Write page content to file:

>>> with open("tmp.html", "w") as dest:
...     dest.writelines(f)
...

Did we get what we expected?

>>> with open("tmp.html") as f: print f.read().split("title")[1]
...
>Python Programming Language &ndash; Official Website</
>>>

[toc] | [prev] | [next] | [standalone]

#8632

Thanks, everyone!
Problem solved.
--
Regards,
Daniil

[toc] | [prev] | [next] | [standalone]

#8633

Hello again!
Another question: urlopen() reads full file's content, but how can I
get page by small parts?

Regards,
Daniil

[toc] | [prev] | [next] | [standalone]

#8639

On Fri, Jul 1, 2011 at 2:23 PM, Даниил Рыжков <daniil.re@gmail.com> wrote:
> Hello again!
> Another question: urlopen() reads full file's content, but how can I
> get page by small parts?
>

Set the Range header for HTTP requests.  The format is specified here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.  Note
that web servers are not *required* to support this header.

In [10]: req = urllib2.Request('http://cdimage.debian.org/debian-cd/6.0.2.1/amd64/iso-cd/debian-6.0.2.1-amd64-CD-1.iso',
headers = { 'Range' : 'bytes=0-499' })

In [11]: f = urllib2.urlopen(req)

In [12]: data = f.read()

In [13]: len(data)
Out[13]: 500

In [14]: print f.headers
Date: Fri, 01 Jul 2011 16:59:39 GMT
Server: Apache/2.2.14 (Unix)
Last-Modified: Sun, 26 Jun 2011 16:54:45 GMT
ETag: "ebff2f-28700000-4a6a04ab27f10"
Accept-Ranges: bytes
Content-Length: 500
Age: 225
Content-Range: bytes 0-499/678428672
Connection: close
Content-Type: application/octet-stream


-- 
regards,
kushal

[toc] | [prev] | [next] | [standalone]

#8640

On Fri, Jul 1, 2011 at 1:53 AM, Даниил Рыжков <daniil.re@gmail.com> wrote:
> Hello again!
> Another question: urlopen() reads full file's content, but how can I
> get page by small parts?

I don't think that's true. Just pass .read() the number of bytes you
want to read, just as you would with an actual file object.

Cheers,
Chris

[toc] | [prev] | [next] | [standalone]

#8651

Thanks, everyone!
Problem solved.

-- 
Regards,
Daniil

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web