Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #93412
| From | dieter <dieter@handshake.de> |
|---|---|
| Subject | Re: Javascript website scraping using WebKit and Selenium tools |
| Date | 2015-07-02 07:48 +0200 |
| References | <mn231m$n5h$1@dont-email.me> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.235.1435816113.3674.python-list@python.org> (permalink) |
Veek M <vek.m1234@gmail.com> writes: > I tried scraping a javascript website using two tools, both didn't work. The > website link is: http://xdguo.taobao.com/category-499399872.htm The relevant > text I'm trying to extract is 'GY-68...': > > <div class="item3line1"> > > <dl class="item " data-id="38952795780"> > <dt class="photo"> > <a target="_blank" href="//item.taobao.com/item.htm?spm=a1z10.5- > c.w4002-6778075404.11.54MDOI&id=38952795780" data-spm-wangpu-module- > id="4002-6778075404" data-spm-anchor-id="a1z10.5-c.w4002-6778075404.11"> > <img > src="//img.alicdn.com/bao/uploaded/i4/TB1HMt3FFXXXXaFaVXXXXXXXXXX_!!0- > item_pic.jpg_240x240.jpg" alt="GY-68 BMP180 ?? BOSCH?? ??????? ?? > BMP085"></img> > </a> > </dt> > ... When I try to access the link above, I am redirected to a login page - which, of course, may look very different from what you expect. You may need to pass on authentication information along with your request in order to get the page you are expecting. Note also, that todays sites often heavily use Javascript - which means that a page only gets the final look when the Javascript has been executed. Once the problems to get the "final" HTML code solved, I would use "lxml" and its "xpath" support to locate any relevant HTML information.
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Javascript website scraping using WebKit and Selenium tools Veek M <vek.m1234@gmail.com> - 2015-07-02 06:41 +0530
Re: Javascript website scraping using WebKit and Selenium tools dieter <dieter@handshake.de> - 2015-07-02 07:48 +0200
Re: Javascript website scraping using WebKit and Selenium tools Veek M <vek.m1234@gmail.com> - 2015-07-02 12:01 +0530
csiph-web