Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #107966 > unrolled thread

You gotta love a 2-line python solution

Back to article view | Back to comp.lang.python

Contents

  You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-01 23:39 -0400
    Re: You gotta love a 2-line python solution Stephen Hansen <me+python@ixokai.io> - 2016-05-01 21:31 -0700
      Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 00:51 -0400
        Re: You gotta love a 2-line python solution Stephen Hansen <me+python@ixokai.io> - 2016-05-01 22:02 -0700
          Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 01:08 -0400
            Re: You gotta love a 2-line python solution Stephen Hansen <me+python@ixokai.io> - 2016-05-01 22:21 -0700
              Re: You gotta love a 2-line python solution Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-05-02 15:51 +1000
          Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 01:23 -0400
            Re: You gotta love a 2-line python solution Stephen Hansen <me+python@ixokai.io> - 2016-05-01 22:37 -0700
              Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 02:13 -0400
    Re: You gotta love a 2-line python solution Terry Reedy <tjreedy@udel.edu> - 2016-05-02 02:46 -0400
    Re: You gotta love a 2-line python solution BartC <bc@freeuk.com> - 2016-05-02 10:26 +0100
      Re: You gotta love a 2-line python solution Marko Rauhamaa <marko@pacujo.net> - 2016-05-02 13:12 +0300
        Re: You gotta love a 2-line python solution Steven D'Aprano <steve@pearwood.info> - 2016-05-02 22:05 +1000
      Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 11:15 -0400
        Re: You gotta love a 2-line python solution Larry Martell <larry.martell@gmail.com> - 2016-05-02 11:24 -0400
        Re: You gotta love a 2-line python solution Manolo Martínez <manolo@austrohungaro.com> - 2016-05-02 17:32 +0200
    Re: You gotta love a 2-line python solution jfong@ms4.hinet.net - 2016-05-02 17:45 -0700
      Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 21:12 -0400
        Re: You gotta love a 2-line python solution jfong@ms4.hinet.net - 2016-05-02 20:27 -0700
          Re: You gotta love a 2-line python solution Stephen Hansen <me+python@ixokai.io> - 2016-05-02 20:49 -0700
            Re: You gotta love a 2-line python solution jfong@ms4.hinet.net - 2016-05-02 20:57 -0700
              Re: You gotta love a 2-line python solution Stephen Hansen <me+python@ixokai.io> - 2016-05-03 09:09 -0700
          Re: You gotta love a 2-line python solution DFS <nospam@dfs.com> - 2016-05-02 23:56 -0400
            Re: You gotta love a 2-line python solution Steven D'Aprano <steve@pearwood.info> - 2016-05-04 11:20 +1000
          Re: You gotta love a 2-line python solution Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2016-05-03 08:14 -0400

Page 1 of 2  [1] 2  Next page →

#107966 — You gotta love a 2-line python solution

To save a webpage to a file:
-------------------------------------
1. import urllib
2. urllib.urlretrieve("http://econpy.pythonanywhere.com
     /ex/001.html","D:\file.html")
-------------------------------------

That's it!

Coming from VB/A background, some of the stuff you can do with python - 
with ease - is amazing.


VBScript version
------------------------------------------------------
1. Option Explicit
2. Dim xmlHTTP, fso, fOut
3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
4. xmlHTTP.Open "GET", "http://econpy.pythonanywhere.com/ex/001.html"
5. xmlHTTP.Send
6. Set fso = CreateObject("Scripting.FileSystemObject")
7. Set fOut = fso.CreateTextFile("D:\file.html", True)
8.  fOut.WriteLine xmlHTTP.ResponseText
9. fOut.Close
10. Set fOut = Nothing
11. Set fso  = Nothing
12. Set xmlHTTP = Nothing
------------------------------------------------------

Technically, that VBS will run with just lines 3-9, but that's still 6 
lines of code vs 2 for python.

[toc] | [next] | [standalone]

#107968

On Sun, May 1, 2016, at 08:39 PM, DFS wrote:
> To save a webpage to a file:
> -------------------------------------
> 1. import urllib
> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>      /ex/001.html","D:\file.html")
> -------------------------------------

Note, for paths on windows you really want to use a rawstring. Ie,
r"D:\file.html".

-- 
Stephen Hansen
  m e @ i x o k a i . i o

[toc] | [prev] | [next] | [standalone]

#107973

On 5/2/2016 12:31 AM, Stephen Hansen wrote:
> On Sun, May 1, 2016, at 08:39 PM, DFS wrote:
>> To save a webpage to a file:
>> -------------------------------------
>> 1. import urllib
>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>      /ex/001.html","D:\file.html")
>> -------------------------------------
>
> Note, for paths on windows you really want to use a rawstring. Ie,
> r"D:\file.html".


Thanks.

I actually use "D:\\file.html" in my code.

[toc] | [prev] | [next] | [standalone]

#107977

On Sun, May 1, 2016, at 09:51 PM, DFS wrote:
> On 5/2/2016 12:31 AM, Stephen Hansen wrote:
> > On Sun, May 1, 2016, at 08:39 PM, DFS wrote:
> >> To save a webpage to a file:
> >> -------------------------------------
> >> 1. import urllib
> >> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
> >>      /ex/001.html","D:\file.html")
> >> -------------------------------------
> >
> > Note, for paths on windows you really want to use a rawstring. Ie,
> > r"D:\file.html".
> > 
> Thanks.
> 
> I actually use "D:\\file.html" in my code.

Or you can do that. But the whole point of raw strings is not having to
escape slashes :) 

-- 
Stephen Hansen
  m e @ i x o k a i . i o

[toc] | [prev] | [next] | [standalone]

#107979

On 5/2/2016 1:02 AM, Stephen Hansen wrote:
> On Sun, May 1, 2016, at 09:51 PM, DFS wrote:
>> On 5/2/2016 12:31 AM, Stephen Hansen wrote:
>>> On Sun, May 1, 2016, at 08:39 PM, DFS wrote:
>>>> To save a webpage to a file:
>>>> -------------------------------------
>>>> 1. import urllib
>>>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>>>      /ex/001.html","D:\file.html")
>>>> -------------------------------------
>>>
>>> Note, for paths on windows you really want to use a rawstring. Ie,
>>> r"D:\file.html".
>>>
>> Thanks.
>>
>> I actually use "D:\\file.html" in my code.
>
> Or you can do that. But the whole point of raw strings is not having to
> escape slashes :)


Nice.  Where/how else is 'r' used?


I'm new to python, but I learned that one the hard way.

I was using "D\testfile.txt" for something, and my code kept failing. 
Took me a while to figure it out.  I tried various letters after the 
slash.  I finally stumbled across the escape slashes in the docs somewhere.

[toc] | [prev] | [next] | [standalone]

#107983

On Sun, May 1, 2016, at 10:08 PM, DFS wrote:
> On 5/2/2016 1:02 AM, Stephen Hansen wrote:
> >> I actually use "D:\\file.html" in my code.
> >
> > Or you can do that. But the whole point of raw strings is not having to
> > escape slashes :)
> 
> 
> Nice.  Where/how else is 'r' used?

Raw strings are primarily used A) for windows paths, and more
universally, B) for regular expressions. 

But in theory they're useful anywhere you have static/literal data that
might include backslashes where you don't actually intend to use any
escape characters.

-- 
Stephen Hansen
  m e @ i x o k a i . i o

[toc] | [prev] | [next] | [standalone]

#107987

On Monday 02 May 2016 15:21, Stephen Hansen wrote:

> On Sun, May 1, 2016, at 10:08 PM, DFS wrote:
>> On 5/2/2016 1:02 AM, Stephen Hansen wrote:
>> >> I actually use "D:\\file.html" in my code.
>> >
>> > Or you can do that. But the whole point of raw strings is not having to
>> > escape slashes :)
>> 
>> 
>> Nice.  Where/how else is 'r' used?
> 
> Raw strings are primarily used A) for windows paths, and more
> universally, B) for regular expressions.

Raw strings are designed for regular expressions. They can be used for 
Windows paths, except for one minor gotcha: you can't end a raw string with 
an odd number of backspaces. So this doesn't work:

directory = r'D:\some\path\dir\'

So it's more of a half-cooked string than a raw string.


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#107984

On 5/2/2016 1:02 AM, Stephen Hansen wrote:
> On Sun, May 1, 2016, at 09:51 PM, DFS wrote:
>> On 5/2/2016 12:31 AM, Stephen Hansen wrote:
>>> On Sun, May 1, 2016, at 08:39 PM, DFS wrote:
>>>> To save a webpage to a file:
>>>> -------------------------------------
>>>> 1. import urllib
>>>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>>>      /ex/001.html","D:\file.html")
>>>> -------------------------------------
>>>
>>> Note, for paths on windows you really want to use a rawstring. Ie,
>>> r"D:\file.html".
>>>
>> Thanks.
>>
>> I actually use "D:\\file.html" in my code.
>
> Or you can do that. But the whole point of raw strings is not having to
> escape slashes :)


Trying the rawstring thing (say it fast 3x):

webpage = "http://econpy.pythonanywhere.com/ex/001.html"

----------------------------------------------------
webfile = "D:\\econpy001.html"
urllib.urlretrieve(webpage,webfile)     WORKS
----------------------------------------------------
webfile = "rD:\econpy001.html"
urllib.urlretrieve(webpage,webfile)     FAILS
----------------------------------------------------
webfile = "D:\econpy001.html"
urllib.urlretrieve(webpage,"r" + webfile)     FAILS
----------------------------------------------------
webfile  = "D:\econpy001.html"
urllib.urlretrieve(webpage,"r" + "" + webfile + "")  FAILS
----------------------------------------------------

The FAILs throw:

Traceback (most recent call last):
   File "webscraper.py", line 54, in <module>
     urllib.urlretrieve(webpage,webfile)
   File "D:\development\python\python_2.7.11\lib\urllib.py", line 98, in 
urlretrieve
     return opener.retrieve(url, filename, reporthook, data)
   File "D:\development\python\python_2.7.11\lib\urllib.py", line 249, 
in retrieve
     tfp = open(filename, 'wb')
IOError: [Errno 22] invalid mode ('wb') or filename: 'rD:\\econpy001.html'
----------------------------------------------------

What am I doing wrong?

[toc] | [prev] | [next] | [standalone]

#107986

On Sun, May 1, 2016, at 10:23 PM, DFS wrote:
> Trying the rawstring thing (say it fast 3x):
> 
> webpage = "http://econpy.pythonanywhere.com/ex/001.html"
> 
> ----------------------------------------------------
> webfile = "D:\\econpy001.html"
> urllib.urlretrieve(webpage,webfile)     WORKS
> ----------------------------------------------------
> webfile = "rD:\econpy001.html"

The r is *outside* the string.

Its: r"D:\econpy001.html"

-- 
Stephen Hansen
  m e @ i x o k a i . i o

[toc] | [prev] | [next] | [standalone]

#107991

On 5/2/2016 1:37 AM, Stephen Hansen wrote:
> On Sun, May 1, 2016, at 10:23 PM, DFS wrote:
>> Trying the rawstring thing (say it fast 3x):
>>
>> webpage = "http://econpy.pythonanywhere.com/ex/001.html"
>>
>> ----------------------------------------------------
>> webfile = "D:\\econpy001.html"
>> urllib.urlretrieve(webpage,webfile)     WORKS
>> ----------------------------------------------------
>> webfile = "rD:\econpy001.html"
>
> The r is *outside* the string.
>
> Its: r"D:\econpy001.html"


Got it.  Thanks.

[toc] | [prev] | [next] | [standalone]

#107995

On 5/2/2016 12:31 AM, Stephen Hansen wrote:
> On Sun, May 1, 2016, at 08:39 PM, DFS wrote:
>> To save a webpage to a file:
>> -------------------------------------
>> 1. import urllib
>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>      /ex/001.html","D:\file.html")
>> -------------------------------------
>
> Note, for paths on windows you really want to use a rawstring. Ie,
> r"D:\file.html".

Or use forward slashes "D:/file.html" and avoid the issue.  I don't know 
of anywhere this does not work for file names sent from python directly 
to Windows.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#108002

On 02/05/2016 04:39, DFS wrote:
> To save a webpage to a file:
> -------------------------------------
> 1. import urllib
> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>     /ex/001.html","D:\file.html")
> -------------------------------------
>
> That's it!
>
> Coming from VB/A background, some of the stuff you can do with python -
> with ease - is amazing.
>
>
> VBScript version
> ------------------------------------------------------
> 1. Option Explicit
> 2. Dim xmlHTTP, fso, fOut
> 3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
> 4. xmlHTTP.Open "GET", "http://econpy.pythonanywhere.com/ex/001.html"
> 5. xmlHTTP.Send
> 6. Set fso = CreateObject("Scripting.FileSystemObject")
> 7. Set fOut = fso.CreateTextFile("D:\file.html", True)
> 8.  fOut.WriteLine xmlHTTP.ResponseText
> 9. fOut.Close
> 10. Set fOut = Nothing
> 11. Set fso  = Nothing
> 12. Set xmlHTTP = Nothing
> ------------------------------------------------------
>
> Technically, that VBS will run with just lines 3-9, but that's still 6
> lines of code vs 2 for python.

It seems Python provides a higher level solution compared with VBS. 
Python presumably also has to do those Opens and Sends, but they are 
hidden away inside urllib.urlretrieve.

You can do the same with VB just by wrapping up these lines in a 
subroutine. As you would if this had to be executed in a dozen different 
places for example. Then you could just write:

getfile("http://econpy.pythonanywhere.com/ex/001.html", "D:/file.html")

in VBS too. (The forward slash in the file name ought to work.)

(I don't know VBS; I assume it does /have/ subroutines? What I haven't 
factored in here is error handling which might yet require more coding 
in VBS compared with Python)

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#108004

BartC <bc@freeuk.com>:

> On 02/05/2016 04:39, DFS wrote:
>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>     /ex/001.html","D:\file.html")
> [...]
>
> It seems Python provides a higher level solution compared with VBS.
> Python presumably also has to do those Opens and Sends, but they are
> hidden away inside urllib.urlretrieve.

Relevant questions include:

 * Is a solution available?

 * Is the solution well thought out?

Python does have a lot of great stuff available, which is nice.
Unfortunately, many of the handy facilities are lacking in the
well-thought-out department.

For example, the urlretrieve() function above blocks. You can't use it
with the asyncio or select modules. You are left with:

   <URL: https://docs.python.org/3/library/asyncio-stream.html#get-http-h
   eaders>

Database facilities are notorious offenders. Also, json.load and
json.loads don't allow you to decode JSON in chunks.

If asyncio breaks through, I expect all blocking stdlib function calls
to be adapted for it over the coming years. I'm not overly fond of the
asyncio programming model, but it does sport two new killer features:

 * any blocking operation can be interrupted

 * events can be multiplexed


Marko

[toc] | [prev] | [next] | [standalone]

#108005

On Mon, 2 May 2016 08:12 pm, Marko Rauhamaa wrote:

> For example, the urlretrieve() function above blocks. You can't use it
> with the asyncio or select modules.


The urlretrieve function is one of the oldest functions in the std library.
It literally only exists because Guido was working on a computer somewhere,
found that he did have wget, and decided it would be faster to write his
own in Python than download and install wget.

And because this was very early in Python's history, the barrier to getting
into the std lib was much less, especially for stuff Guido wrote himself,
so there it is. These days, I doubt it would be included. It would probably
be a recipe in the docs.

Compared to a full-featured tool like wget or curl, urlretrieve is missing a
lot of stuff which is considered essential, like limiting/configuring the
rate, support for cookies and authentication, retrying on error, etc.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#108014

On 5/2/2016 5:26 AM, BartC wrote:
> On 02/05/2016 04:39, DFS wrote:
>> To save a webpage to a file:
>> -------------------------------------
>> 1. import urllib
>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>     /ex/001.html","D:\file.html")
>> -------------------------------------
>>
>> That's it!
>>
>> Coming from VB/A background, some of the stuff you can do with python -
>> with ease - is amazing.
>>
>>
>> VBScript version
>> ------------------------------------------------------
>> 1. Option Explicit
>> 2. Dim xmlHTTP, fso, fOut
>> 3. Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
>> 4. xmlHTTP.Open "GET", "http://econpy.pythonanywhere.com/ex/001.html"
>> 5. xmlHTTP.Send
>> 6. Set fso = CreateObject("Scripting.FileSystemObject")
>> 7. Set fOut = fso.CreateTextFile("D:\file.html", True)
>> 8.  fOut.WriteLine xmlHTTP.ResponseText
>> 9. fOut.Close
>> 10. Set fOut = Nothing
>> 11. Set fso  = Nothing
>> 12. Set xmlHTTP = Nothing
>> ------------------------------------------------------
>>
>> Technically, that VBS will run with just lines 3-9, but that's still 6
>> lines of code vs 2 for python.
>
> It seems Python provides a higher level solution compared with VBS.
> Python presumably also has to do those Opens and Sends, but they are
> hidden away inside urllib.urlretrieve.
>
> You can do the same with VB just by wrapping up these lines in a
> subroutine. As you would if this had to be executed in a dozen different
> places for example. Then you could just write:
>
> getfile("http://econpy.pythonanywhere.com/ex/001.html", "D:/file.html")
>
> in VBS too. (The forward slash in the file name ought to work.)


Of course.  Taken to its extreme, I could eventually replace you with 
one line of code :)

But python does it for me.  That would save me 8 lines...



> (I don't know VBS; I assume it does /have/ subroutines? What I haven't
> factored in here is error handling which might yet require more coding
> in VBS compared with Python)

Yeah, VBS has subs and functions.  And strange, limited error handling. 
And a single data type, called Variant.  But it's installed with Windows 
so it's easy to get going with.

[toc] | [prev] | [next] | [standalone]

#108015

On Mon, May 2, 2016 at 11:15 AM, DFS <nospam@dfs.com> wrote:
> Of course.  Taken to its extreme, I could eventually replace you with one
> line of code :)

That reminds me of something I heard many years ago.

Every non-trivial program can be simplified by at least one line of code.
Every non trivial program has at least one bug.

Therefore every non-trivial program can be reduced to one line of code
with a bug.

[toc] | [prev] | [next] | [standalone]

#108017

On 05/02/16 at 11:24am, Larry Martell wrote:
> That reminds me of something I heard many years ago.
> 
> Every non-trivial program can be simplified by at least one line of code.
> Every non trivial program has at least one bug.
> 
> Therefore every non-trivial program can be reduced to one line of code
> with a bug.

Well, not really. Every non-trivial program can be reduced to one line
of code, but then the resulting program is not non-trivial (as it cannot
be further reduced), and therefore there are no guarantees that it will
have a bug.

M

[toc] | [prev] | [next] | [standalone]

#108037

DFS at 2016/5/2 UTC+8 11:39:33AM wrote:
> To save a webpage to a file:
> -------------------------------------
> 1. import urllib
> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>      /ex/001.html","D:\file.html")
> -------------------------------------
> 
> That's it!

Why my system can't do it?

Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib import urlretrieve
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'urlretrieve'

[toc] | [prev] | [next] | [standalone]

#108038

On 5/2/2016 8:45 PM, jfong@ms4.hinet.net wrote:
> DFS at 2016/5/2 UTC+8 11:39:33AM wrote:
>> To save a webpage to a file:
>> -------------------------------------
>> 1. import urllib
>> 2. urllib.urlretrieve("http://econpy.pythonanywhere.com
>>      /ex/001.html","D:\file.html")
>> -------------------------------------
>>
>> That's it!
>
> Why my system can't do it?
>
> Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
> tel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from urllib import urlretrieve
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: cannot import name 'urlretrieve'


try

from urllib.request import urlretrieve

http://stackoverflow.com/questions/21171718/urllib-urlretrieve-file-python-3-3


I'm running python 2.7.11 (32-bit)

[toc] | [prev] | [next] | [standalone]

#108043

DFS at 2016/5/3 9:12:24AM wrote:
> try
> 
> from urllib.request import urlretrieve
> 
> http://stackoverflow.com/questions/21171718/urllib-urlretrieve-file-python-3-3
> 
> 
> I'm running python 2.7.11 (32-bit)

Alright, it works...someway.

I try to get a zip file. It works, the file can be unzipped correctly.

>>> from urllib.request import urlretrieve
>>> urlretrieve("http://www.caprilion.com.tw/fed.zip", "d:\\temp\\temp.zip")
('d:\\temp\\temp.zip', <http.client.HTTPMessage object at 0x03102C50>)
>>>

But when I try to get this forum page, it does get a html file but can't be viewed normally.

>>> urlretrieve("https://groups.google.com/forum/#!topic/comp.lang.python/jFl3GJ
bmR7A", "d:\\temp\\temp.html")
('d:\\temp\\temp.html', <http.client.HTTPMessage object at 0x03102A90>)
>>>

I suppose the html is a much complex situation where more processes need to be done before it can be opened by a web browser:-)

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web