Path: csiph.com!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail From: Robert Klemme Newsgroups: comp.lang.ruby Subject: Re: HTTP response question. Date: Sun, 20 Dec 2015 12:03:28 +0100 Lines: 86 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Trace: individual.net N9psWcW9CGFEQrP4KBZOaQLCNcrlqAhzK8V/cZh49Qrf4P05o= Cancel-Lock: sha1:5k+h7JvkiXlyrnA2F22Q/oSyLHk= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 In-Reply-To: Xref: csiph.com comp.lang.ruby:7167 On 20.12.2015 09:58, Michael Uplawski wrote: > In one program, I receive a Content-Length header and am able to resume > partial downloads, in the second, the Content-Length header appears to > always be *missing*. > The program which is always successful is restricted to the retrieval of > one type of data from two precise web-sites. Put another way: I do not > know a lot about HTTP and learn much by trial and error (and some hints > from Mr. Klemm on this newsgroup, of course). Alas, my trials with the > second program are all futile, so far. > > Question: Is it me or is it the responding site? Can I do anything > better to resume the previously interrupted download? There is no guarantee that a server will send Content-Length header. As simple as that. You can check with curl: HEAD request $ url='http://www.fosshub.com/IrfanView.html/iview441_setup.exe' $ curl --head -D headers.txt -o /dev/null "$url" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 $ cat headers.txt HTTP/1.1 200 OK Date: Sun, 20 Dec 2015 10:57:56 GMT Content-Type: text/html; charset=utf-8 Connection: keep-alive Set-Cookie: __cfduid=d5dc7775a3bc4742413be867fb87e18ae1450609075; expires=Mon, 19-Dec-16 10:57:55 GMT; path=/; domain=.fosshub.com; HttpOnly Vary: Accept-Encoding X-Powered-By: PHP/5.4.36 Age: 59104 Via: 1.1 varnish X-Cache: HIT X-Cache-Hits: 726 Server: cloudflare-nginx CF-RAY: 257ad444c16b271a-FRA GET request: $ curl --get -D headers.txt -o /dev/null "$url" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 46352 0 46352 0 0 114k 0 --:--:-- --:--:-- --:--:-- 114k $ cat headers.txt HTTP/1.1 200 OK Date: Sun, 20 Dec 2015 10:58:42 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive Set-Cookie: __cfduid=da523a3a1942bb307a1e7ed6f847ee36a1450609122; expires=Mon, 19-Dec-16 10:58:42 GMT; path=/; domain=.fosshub.com; HttpOnly Vary: Accept-Encoding X-Powered-By: PHP/5.4.36 Age: 59151 Via: 1.1 varnish X-Cache: HIT X-Cache-Hits: 727 Server: cloudflare-nginx CF-RAY: 257ad5684e292684-FRA Note, there is some line wrapping interfering but you can try out yourself. Regarding your code: first thing you should probably do is to check the HTTP status code to avoid downloading a HTML error page for the binary that you are expecting. You could also use Mechanize to take care of session and cookie handling. Cheers robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/