Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #107967 > unrolled thread

Fastest way to retrieve and write html contents to file

Back to article view | Back to comp.lang.python

Contents

  Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 00:06 -0400
    Re: Fastest way to retrieve and write html contents to file Stephen Hansen <me+python@ixokai.io> - 2016-05-01 21:34 -0700
    Re: Fastest way to retrieve and write html contents to file Chris Angelico <rosuav@gmail.com> - 2016-05-02 14:40 +1000
      Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 00:50 -0400
        Re: Fastest way to retrieve and write html contents to file Stephen Hansen <me+python@ixokai.io> - 2016-05-01 22:00 -0700
          Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 01:04 -0400
            Re: Fastest way to retrieve and write html contents to file Chris Angelico <rosuav@gmail.com> - 2016-05-02 15:12 +1000
            Re: Fastest way to retrieve and write html contents to file Stephen Hansen <me+python@ixokai.io> - 2016-05-01 22:17 -0700
            Re: Fastest way to retrieve and write html contents to file Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-05-02 15:57 +1000
    Re: Fastest way to retrieve and write html contents to file Ben Finney <ben+python@benfinney.id.au> - 2016-05-02 14:49 +1000
      Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 01:00 -0400
        Re: Fastest way to retrieve and write html contents to file Stephen Hansen <me+python@ixokai.io> - 2016-05-01 22:15 -0700
          Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 01:59 -0400
            Re: Fastest way to retrieve and write html contents to file Stephen Hansen <me+python@ixokai.io> - 2016-05-01 23:27 -0700
              Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 03:37 -0400
                Re: Fastest way to retrieve and write html contents to file Stephen Hansen <me+python@ixokai.io> - 2016-05-02 00:58 -0700
                Re: Fastest way to retrieve and write html contents to file Michael Torrie <torriem@gmail.com> - 2016-05-02 22:06 -0600
                  Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-03 00:24 -0400
                    Re: Fastest way to retrieve and write html contents to file Tim Chase <python.list@tim.thechases.com> - 2016-05-03 10:28 -0500
                      Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-03 13:00 -0400
                        Re: Fastest way to retrieve and write html contents to file Tim Chase <python.list@tim.thechases.com> - 2016-05-03 13:41 -0500
                          Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-04 02:10 -0400
        Re: Fastest way to retrieve and write html contents to file Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-05-02 16:05 +1000
          Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 02:47 -0400
            Re: Fastest way to retrieve and write html contents to file Chris Angelico <rosuav@gmail.com> - 2016-05-02 17:19 +1000
              Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 21:51 -0400
                Re: Fastest way to retrieve and write html contents to file Chris Angelico <rosuav@gmail.com> - 2016-05-03 12:00 +1000
                  Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 22:01 -0400
            Re: Fastest way to retrieve and write html contents to file Peter Otten <__peter__@web.de> - 2016-05-02 10:42 +0200
              Re: Fastest way to retrieve and write html contents to file DFS <nospam@dfs.com> - 2016-05-02 21:52 -0400
    Re: Fastest way to retrieve and write html contents to file Chris Angelico <rosuav@gmail.com> - 2016-05-02 14:53 +1000
    Re: Fastest way to retrieve and write html contents to file Tim Chase <python.list@tim.thechases.com> - 2016-05-02 07:38 -0500

Page 2 of 2 — ← Prev page 1 [2]

#108099

On 2016-05-03 13:00, DFS wrote:
> On 5/3/2016 11:28 AM, Tim Chase wrote:
> > On 2016-05-03 00:24, DFS wrote:
> >> One small comparison I was able to make was VBA vs python/pyodbc
> >> to summarize an Access database.  Not quite a fair test, but
> >> interesting nonetheless.
> >>
> >> Access 2003 file
> >> Access 2003 VBA code
> >> Time: 0.18 seconds
> >>
> >> same Access 2003 file
> >> 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6
> >> Time: 0.49 seconds
> >
> > Curious whether you're forcing Access VBA to talk over ODBC or
> > whether Access is using native access/file-handling (and thus
> > bypassing the ODBC overhead)?
> 
> The latter, which is why I said "not quite a fair test".

Can you try the same tests, getting Access/VBA to use ODBC instead to
see how much overhead ODBC entails?

-tkc

[toc] | [prev] | [next] | [standalone]

#108116

On 5/3/2016 2:41 PM, Tim Chase wrote:
> On 2016-05-03 13:00, DFS wrote:
>> On 5/3/2016 11:28 AM, Tim Chase wrote:
>>> On 2016-05-03 00:24, DFS wrote:
>>>> One small comparison I was able to make was VBA vs python/pyodbc
>>>> to summarize an Access database.  Not quite a fair test, but
>>>> interesting nonetheless.
>>>>
>>>> Access 2003 file
>>>> Access 2003 VBA code
>>>> Time: 0.18 seconds
>>>>
>>>> same Access 2003 file
>>>> 32-bit python 2.7.11 + 32-bit pyodbc 3.0.6
>>>> Time: 0.49 seconds
>>>
>>> Curious whether you're forcing Access VBA to talk over ODBC or
>>> whether Access is using native access/file-handling (and thus
>>> bypassing the ODBC overhead)?
>>
>> The latter, which is why I said "not quite a fair test".
>
> Can you try the same tests, getting Access/VBA to use ODBC instead to
> see how much overhead ODBC entails?
>
> -tkc


Done.

I dropped a few extraneous tables from the database (was 114 tables):

Access 2003 .mdb file
2,009,164 rows
97 tables  (max row = 600288)
725 columns
   text:      389
   boolean:   4
   numeric:   261
   date-time: 69
   binary:    2
264 indexes (25 foreign keys)*
299,167,744 bytes on disk


1. DAO
    Time: 0.15 seconds

2. ADODB, Access ODBC driver, OpenSchema method**
    Time: 0.26 seconds

3. python, pyodbc, Access ODBC driver
    Time: 0.42 seconds




* despite being written by Microsoft, the Access ODBC driver doesn't
   support the ODBC SQLForeignKeys function, so the python code doesn't
   show a count of foreign keys

** the Access ODBC driver doesn't support the adSchemaIndexes or
    adSchemaForeignKeys query types, so I used DAO code to count
    indexes and foreign keys.

[toc] | [prev] | [next] | [standalone]

#107990

On Monday 02 May 2016 15:00, DFS wrote:

> I tried the 10-loop test several times with all versions.
> 
> The results were 100% consistent: VBSCript xmlHTTP was always 2x faster
> than any python method.


Are you absolutely sure you're comparing the same job in two languages? Is 
VB using a local web cache, and Python not? Are you saving files with both 
tests? To the same local drive? (To ensure you aren't measuring the 
difference between "write this file to a slow IDE hard disk, write that file 
to a fast SSD".)

Once you are sure that you are comparing the same task in two languages, 
then make sure the measurement is meaningful. If you change from a (let's 
say) 1 KB file to a 100 KB file, do you see the same 2 x difference? What if 
you increase it to a 10000 KB file?


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#107996

On 5/2/2016 2:05 AM, Steven D'Aprano wrote:
> On Monday 02 May 2016 15:00, DFS wrote:
>
>> I tried the 10-loop test several times with all versions.
>>
>> The results were 100% consistent: VBSCript xmlHTTP was always 2x faster
>> than any python method.
>
>
> Are you absolutely sure you're comparing the same job in two languages?

As near as I can tell.  In VBScript I'm actually dereferencing various 
objects (that adds to the time), but I don't do that in python.  I don't 
know enough to even know if it's necessary, or good practice, or what.




> Is VB using a local web cache, and Python not?

I'm not specifying a local web cache with either (wouldn't know how or 
where to look).  If you have Windows, you can try it.
-------------------------------------------------------------------
Option Explicit
Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i
webpage = "http://econpy.pythonanywhere.com/ex/001.html"
webfile  = "D:\econpy001.html"
startTime = Timer
For i = 1 to 10
  Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
  xmlHTTP.Open "GET", webpage
  xmlHTTP.Send
  Set fso = CreateObject("Scripting.FileSystemObject")
  Set fOut = fso.CreateTextFile(webfile, True)
   fOut.WriteLine xmlHTTP.ResponseText
  fOut.Close
  Set fOut    = Nothing
  Set fso     = Nothing
  Set xmlHTTP = Nothing
Next
endTime = Timer
wscript.echo "Finished VBScript in " & FormatNumber(endTime - 
startTime,3) & " seconds"
-------------------------------------------------------------------
save it to a .vbs file and run it like this:
$cscript /nologo filename.vbs


> Are you saving files with both
> tests? To the same local drive? (To ensure you aren't measuring the
> difference between "write this file to a slow IDE hard disk, write that file
> to a fast SSD".)

Identical functionality (retrieve webpage, write html to file).  Same 
webpage, written to the same folder on the same hard drive (not SSD).

The 10 file writes (open/write/close) don't make a meaningful difference 
at all:
VBScript 0.0156 seconds
urllib2  0.0034 seconds

This file is 3.55K.


> Once you are sure that you are comparing the same task in two languages,
> then make sure the measurement is meaningful. If you change from a (let's
> say) 1 KB file to a 100 KB file, do you see the same 2 x difference? What if
> you increase it to a 10000 KB file?

Do you know a webpage I can hit 10x repeatedly to download a good size 
file?  I'm always paranoid they'll block me thinking I'm a 
"professional" web scraper or something.

Thanks

[toc] | [prev] | [next] | [standalone]

#107997

On Mon, May 2, 2016 at 4:47 PM, DFS <nospam@dfs.com> wrote:
> I'm not specifying a local web cache with either (wouldn't know how or where
> to look).  If you have Windows, you can try it.
> -------------------------------------------------------------------
> Option Explicit
> Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i
> webpage = "http://econpy.pythonanywhere.com/ex/001.html"
> webfile  = "D:\econpy001.html"
> startTime = Timer
> For i = 1 to 10
>  Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
>  xmlHTTP.Open "GET", webpage
>  xmlHTTP.Send
>  Set fso = CreateObject("Scripting.FileSystemObject")
>  Set fOut = fso.CreateTextFile(webfile, True)
>   fOut.WriteLine xmlHTTP.ResponseText
>  fOut.Close
>  Set fOut    = Nothing
>  Set fso     = Nothing
>  Set xmlHTTP = Nothing
> Next
> endTime = Timer
> wscript.echo "Finished VBScript in " & FormatNumber(endTime - startTime,3) &
> " seconds"
> -------------------------------------------------------------------

There's an easier way to test if there's caching happening. Just crank
the iterations up from 10 to 100 and see what happens to the times. If
your numbers are perfectly fair, they should be perfectly linear in
the iteration count; eg a 1.8 second ten-iteration loop should become
an 18 second hundred-iteration loop. Obviously they won't be exactly
that, but I would expect them to be reasonably close (eg 17-19
seconds, but not 2 seconds).

Then the next thing to test would be to create a deliberately-slow web
server, and connect to that. Put a two-second delay into it, to
simulate a distant or overloaded server, and see if your logs show the
correct result. Something like this:

--------

import time
try:
    import http.server as BaseHTTPServer # Python 3
except ImportError:
    import BaseHTTPServer # Python 2

class SlowHTTP(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type","text/html")
        self.end_headers()
        self.wfile.write(b"Hello, ")
        time.sleep(2)
        self.wfile.write(b"world!")

server = BaseHTTPServer.HTTPServer(("", 1234), SlowHTTP)
server.serve_forever()

-------

Test that with a web browser or command-line downloader (go to
http://127.0.0.1:1234/), and make sure that (a) it produces "Hello,
world!", and (b) it takes two seconds. Then set your test scripts to
downloading that URL. (Be sure to set them back to low iteration
counts first!) If the times are true and fair, they should all come
out pretty much the same - ten iterations, twenty seconds. And since
all that's changed is the server, this will be an accurate
demonstration of what happens in the real world: network requests
aren't always fast. Incidentally, you can also watch the server's log
to see if it's getting the appropriate number of requests.

It may turn out that changing the web server actually materially
changes your numbers. Comment out the sleep call and try it again -
you might find that your numbers come closer together, because this
naive server doesn't send back 204 NOT MODIFIED responses or anything.
Again, though, this would prove that you're not actually measuring
language performance, because the tests are more dependent on the
server than the client.

Even if the files themselves aren't being cached, you might find that
DNS is. So if you truly want to eliminate variables, replace the name
in your URL with an IP address. It's another thing that might mess
with your timings, without actually being a language feature.

Networking has about four billion variables in it. You're messing with
one of the least significant: the programming language :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#108039

On 5/2/2016 3:19 AM, Chris Angelico wrote:

> There's an easier way to test if there's caching happening. Just crank
> the iterations up from 10 to 100 and see what happens to the times. If
> your numbers are perfectly fair, they should be perfectly linear in
> the iteration count; eg a 1.8 second ten-iteration loop should become
> an 18 second hundred-iteration loop. Obviously they won't be exactly
> that, but I would expect them to be reasonably close (eg 17-19
> seconds, but not 2 seconds).

100 loops
Finished VBScript in 3.953 seconds
Finished VBScript in 3.608 seconds
Finished VBScript in 3.610 seconds

Bit of a per-loop speedup going from 10 to 100.


> Then the next thing to test would be to create a deliberately-slow web
> server, and connect to that. Put a two-second delay into it, to
> simulate a distant or overloaded server, and see if your logs show the
> correct result. Something like this:
>
> --------
>
> import time
> try:
>     import http.server as BaseHTTPServer # Python 3
> except ImportError:
>     import BaseHTTPServer # Python 2
>
> class SlowHTTP(BaseHTTPServer.BaseHTTPRequestHandler):
>     def do_GET(self):
>         self.send_response(200)
>         self.send_header("Content-type","text/html")
>         self.end_headers()
>         self.wfile.write(b"Hello, ")
>         time.sleep(2)
>         self.wfile.write(b"world!")
>
> server = BaseHTTPServer.HTTPServer(("", 1234), SlowHTTP)
> server.serve_forever()
>
> -------
>
> Test that with a web browser or command-line downloader (go to
> http://127.0.0.1:1234/), and make sure that (a) it produces "Hello,
> world!", and (b) it takes two seconds. Then set your test scripts to
> downloading that URL. (Be sure to set them back to low iteration
> counts first!) If the times are true and fair, they should all come
> out pretty much the same - ten iterations, twenty seconds. And since
> all that's changed is the server, this will be an accurate
> demonstration of what happens in the real world: network requests
> aren't always fast. Incidentally, you can also watch the server's log
> to see if it's getting the appropriate number of requests.
>
> It may turn out that changing the web server actually materially
> changes your numbers. Comment out the sleep call and try it again -
> you might find that your numbers come closer together, because this
> naive server doesn't send back 204 NOT MODIFIED responses or anything.
> Again, though, this would prove that you're not actually measuring
> language performance, because the tests are more dependent on the
> server than the client.
>
> Even if the files themselves aren't being cached, you might find that
> DNS is. So if you truly want to eliminate variables, replace the name
> in your URL with an IP address. It's another thing that might mess
> with your timings, without actually being a language feature.
>
> Networking has about four billion variables in it. You're messing with
> one of the least significant: the programming language :)
>
> ChrisA


Thanks for the good feedback.

[toc] | [prev] | [next] | [standalone]

#108041

On Tue, May 3, 2016 at 11:51 AM, DFS <nospam@dfs.com> wrote:
> On 5/2/2016 3:19 AM, Chris Angelico wrote:
>
>> There's an easier way to test if there's caching happening. Just crank
>> the iterations up from 10 to 100 and see what happens to the times. If
>> your numbers are perfectly fair, they should be perfectly linear in
>> the iteration count; eg a 1.8 second ten-iteration loop should become
>> an 18 second hundred-iteration loop. Obviously they won't be exactly
>> that, but I would expect them to be reasonably close (eg 17-19
>> seconds, but not 2 seconds).
>
>
> 100 loops
> Finished VBScript in 3.953 seconds
> Finished VBScript in 3.608 seconds
> Finished VBScript in 3.610 seconds
>
> Bit of a per-loop speedup going from 10 to 100.

How many seconds was it for 10 loops?

ChrisA

[toc] | [prev] | [next] | [standalone]

#108042

On 5/2/2016 10:00 PM, Chris Angelico wrote:
> On Tue, May 3, 2016 at 11:51 AM, DFS <nospam@dfs.com> wrote:
>> On 5/2/2016 3:19 AM, Chris Angelico wrote:
>>
>>> There's an easier way to test if there's caching happening. Just crank
>>> the iterations up from 10 to 100 and see what happens to the times. If
>>> your numbers are perfectly fair, they should be perfectly linear in
>>> the iteration count; eg a 1.8 second ten-iteration loop should become
>>> an 18 second hundred-iteration loop. Obviously they won't be exactly
>>> that, but I would expect them to be reasonably close (eg 17-19
>>> seconds, but not 2 seconds).
>>
>>
>> 100 loops
>> Finished VBScript in 3.953 seconds
>> Finished VBScript in 3.608 seconds
>> Finished VBScript in 3.610 seconds
>>
>> Bit of a per-loop speedup going from 10 to 100.
>
> How many seconds was it for 10 loops?
>
> ChrisA

~0.44

[toc] | [prev] | [next] | [standalone]

#108001

DFS wrote:

>> Is VB using a local web cache, and Python not?
> 
> I'm not specifying a local web cache with either (wouldn't know how or
> where to look).  If you have Windows, you can try it.

I don't have Windows, but if I'm to believe

http://stackoverflow.com/questions/5235464/how-to-make-microsoft-xmlhttprequest-honor-cache-control-directive

the page is indeed cached and you can disable caching with

> Option Explicit
> Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i
> webpage = "http://econpy.pythonanywhere.com/ex/001.html"
> webfile  = "D:\econpy001.html"
> startTime = Timer
> For i = 1 to 10
> Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
> xmlHTTP.Open "GET", webpage
  
  xmlHTTP.setRequestHeader "Cache-Control", "max-age=0"

> xmlHTTP.Send
> Set fso = CreateObject("Scripting.FileSystemObject")
> Set fOut = fso.CreateTextFile(webfile, True)
> fOut.WriteLine xmlHTTP.ResponseText
> fOut.Close
> Set fOut    = Nothing
> Set fso     = Nothing
> Set xmlHTTP = Nothing
> Next
> endTime = Timer
> wscript.echo "Finished VBScript in " & FormatNumber(endTime -
> startTime,3) & " seconds"
> -------------------------------------------------------------------
> save it to a .vbs file and run it like this:
> $cscript /nologo filename.vbs
> 

[toc] | [prev] | [next] | [standalone]

#108040

On 5/2/2016 4:42 AM, Peter Otten wrote:
> DFS wrote:
>
>>> Is VB using a local web cache, and Python not?
>>
>> I'm not specifying a local web cache with either (wouldn't know how or
>> where to look).  If you have Windows, you can try it.
>
> I don't have Windows, but if I'm to believe
>
> http://stackoverflow.com/questions/5235464/how-to-make-microsoft-xmlhttprequest-honor-cache-control-directive
>
> the page is indeed cached and you can disable caching with
>
>> Option Explicit
>> Dim xmlHTTP, fso, fOut, startTime, endTime, webpage, webfile,i
>> webpage = "http://econpy.pythonanywhere.com/ex/001.html"
>> webfile  = "D:\econpy001.html"
>> startTime = Timer
>> For i = 1 to 10
>> Set xmlHTTP = CreateObject("MSXML2.serverXMLHTTP")
>> xmlHTTP.Open "GET", webpage
>
>   xmlHTTP.setRequestHeader "Cache-Control", "max-age=0"


Tried that, and from later on that stackoverflow page:

xmlHTTP.setRequestHeader "Cache-Control", "private"

Neither made a difference.  In fact, I saw faster times than ever - as 
low as 0.41 for 10 loops.

[toc] | [prev] | [next] | [standalone]

#107974

On Mon, May 2, 2016 at 2:49 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
> One simple way to do that: Run the exact same test many times (say,
> 10 000 or so) on the same machine, and then compute the average of all
> the durations.
>
> Do the same for each different program, and then you may have more
> meaningfully comparable measurements.

And also find the minimum and maximum durations, too. Averages don't
always tell the whole story.

ChrisA

[toc] | [prev] | [next] | [standalone]

#108006

On 2016-05-02 00:06, DFS wrote:
> Then I tested them in loops - the VBScript is MUCH faster: 0.44 for
> 10 iterations, vs 0.88 for python.

In addition to the other debugging recommendations in sibling
threads, a couple other things to try:

1) use a local debugging proxy so that you can compare the headers to
see if anything stands out

2) in light of #1, can you confirm/deny whether one is using gzip
compression and the other isn't?

-tkc

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web