Re: simple download manager

Path	csiph.com!usenet.pasdenom.info!gegeweb.org!news.ecp.fr!news.stack.nl!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<rosuav@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.013
X-Spam-Evidence	'H': 0.98; 'S': 0.00; 'python.': 0.02; 'automate': 0.07; 'javascript,': 0.07; 'buttons': 0.09; 'chunk': 0.09; 'http': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; '(just': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'pulling': 0.16; 'subject:download': 0.16; 'subject:simple': 0.16; 'through.': 0.16; 'wrote:': 0.18; 'library': 0.18; 'wed,': 0.18; 'code,': 0.22; 'cc:addr:python.org': 0.22; 'cc:2**0': 0.24; 'query': 0.26; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'message- id:@mail.gmail.com': 0.30; 'figure': 0.32; 'service,': 0.32; 'could': 0.34; 'but': 0.35; 'received:google.com': 0.35; 'google': 0.35; 'really': 0.36; 'should': 0.36; 'easily': 0.37; 'nov': 0.38; 'tools,': 0.38; 'handle': 0.38; 'files': 0.38; 'how': 0.40; 'simple': 0.61; "you're": 0.61; 'watching': 0.68; 'wish': 0.70; 'click': 0.77; 'hand': 0.80; 'off,': 0.84; 'replicate': 0.84; "site's": 0.84; 'that),': 0.91; 'to:none': 0.92; 'ads': 0.97
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=/DYu7DBw3DrOfpLrJH/GNh/Dyk3fIxyogLhLfpBgRLM=; b=L4Zz06ThnvdCOh5zlHKxlfInrJJGRbjurQrwH11598/WSPMN/vojs6RRP2p45b6sgV A7och1HrWHNlsQzDEA8LaFuP6HlGBbO+seCmREBf3ssZ1V+8LYmmQLoVCOw6bqC41cYb 9qtOQ1pzJEmyJf8P+lbJzeMZ0PI0TvAQ6on4yvrkbNdNRxgtC1Xoc64u3IHe0hMijerl 4NvEalJFerEVAUEBqssezLm/vvFKFguuHqSwtHS2sXgsNgvAX0PHCq6BTFB4HdggDdZ7 fbV+VH3HIUfHy2n/Vf+mjdJQsf0CrHXme6JJfzVghbkM4xZR0HK+tFzhmX8gPzuLiYAl xP8A==
MIME-Version	1.0
X-Received	by 10.107.136.94 with SMTP id k91mr39592774iod.43.1415113193012; Tue, 04 Nov 2014 06:59:53 -0800 (PST)
In-Reply-To	<5d9b7957-ee57-4cc4-b4eb-7a933bad1a5e@googlegroups.com>
References	<5d9b7957-ee57-4cc4-b4eb-7a933bad1a5e@googlegroups.com>
Date	Wed, 5 Nov 2014 01:59:52 +1100
Subject	Re: simple download manager
From	Chris Angelico <rosuav@gmail.com>
Cc	"python-list@python.org" <python-list@python.org>
Content-Type	text/plain; charset=UTF-8
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.15449.1415113202.18130.python-list@python.org> (permalink)
Lines	19
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1415113202 news.xs4all.nl 2925 [2001:888:2000:d::a6]:54531
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:80190

Show key headers only | View raw

On Wed, Nov 5, 2014 at 1:53 AM, Kiuhnm <gandalf23@mail.com> wrote:
> I wish to automate the downloading from a particular site which has some ADs and which requires to click on a lot of buttons before the download starts.
>
> What library should I use to handle HTTP?
> Also, I need to support big files (> 1 GB) so the library should hand the data to me chunk by chunk.

You may be violating the site's terms of service, so be aware of what
you're doing.

This could be a really simple job (just figure out what the last HTTP
query is, and replicate that), or it could be insanely complicated
(crypto, JavaScript, and/or timestamped URLs could easily be
involved). To start off, I would recommend not writing a single like
of Python code, but just pulling up Mozilla Firefox with Firebug, or
Google Chrome with in-built inspection tools, or some equivalent, and
watching the exact queries that go through. Once you figure out what
queries are happening, you can figure out how to do them in Python.

ChrisA

Thread

simple download manager Kiuhnm <gandalf23@mail.com> - 2014-11-04 06:53 -0800
  Re: simple download manager Chris Angelico <rosuav@gmail.com> - 2014-11-05 01:59 +1100
    Re: simple download manager Kiuhnm <gandalf23@mail.com> - 2014-11-04 07:10 -0800
      Re: simple download manager Kiuhnm <gandalf23@mail.com> - 2014-11-04 16:38 -0800

csiph-web