Re: Python3 html scraper that supports javascript

From	Stephen Hansen <me+python@ixokai.io>
Newsgroups	comp.lang.python
Subject	Re: Python3 html scraper that supports javascript
Date	2016-05-02 11:00 -0700
Message-ID	<mailman.327.1462212027.32212.python-list@python.org> (permalink)
References	<2a0c92ed-352d-455c-832d-c9a9438f318b@googlegroups.com> <CAP1rxO5_ypuejA3eXowPKcndsyAy1k9CzCdO8L5KWuqsK1-X1g@mail.gmail.com> <mailman.285.1462122077.32212.python-list@python.org> <d8db7fec-0083-44ef-8f5b-73d097789b9b@googlegroups.com> <1462212024.1321698.595856209.333A021A@webmail.messagingengine.com>

Show all headers | View raw

On Mon, May 2, 2016, at 08:33 AM, zljubisic@gmail.com wrote:
> I tried to use the following code:
> 
> from bs4 import BeautifulSoup
> from selenium import webdriver
> 
> PHANTOMJS_PATH =
> 'C:\\Users\\Zoran\\Downloads\\Obrisi\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe'
> 
> url =
> 'https://hrti.hrt.hr/#/video/show/2203605/trebizat-prica-o-jednoj-vodi-i-jednom-narodu-dokumentarni-film'
> 
> browser = webdriver.PhantomJS(PHANTOMJS_PATH)
> browser.get(url)
> 
> soup = BeautifulSoup(browser.page_source, "html.parser")
> 
> x = soup.prettify()
> 
> print(x)
> 
> When I print x variable, I would expect to see something like this:
> <video
> src="mediasource:https://hrti.hrt.hr/2e9e9c45-aa23-4d08-9055-cd2d7f2c4d58"
> id="vjs_video_3_html5_api" class="vjs-tech" preload="none"><source
> type="application/x-mpegURL"
> src="https://prd-hrt.spectar.tv/player/get_smil/id/2203605/video_id/2203605/token/Cny6ga5VEQSJ2uZaD2G8pg/token_expiration/1462043309/asset_type/Movie/playlist_template/nginx/channel_name/trebiat__pria_o_jednoj_vodi_i_jednom_narodu_dokumentarni_film/playlist.m3u8?foo=bar">
> </video>
> 
> but I can't come to that point.

Why? As important as it is to show code, you need to show what actually
happens and what error message is produced.

-- 
Stephen Hansen
  m e @ i x o k a i . i o

Thread

Python3 html scraper that supports javascript zljubisic@gmail.com - 2016-05-01 07:19 -0700
  Re: Python3 html scraper that supports javascript Bob Gailer <bgailer@gmail.com> - 2016-05-01 13:01 -0400
    Re: Python3 html scraper that supports javascript zljubisic@gmail.com - 2016-05-02 08:33 -0700
      Re: Python3 html scraper that supports javascript DFS <nospam@dfs.com> - 2016-05-02 12:39 -0400
      Re: Python3 html scraper that supports javascript Stephen Hansen <me+python@ixokai.io> - 2016-05-02 11:00 -0700
        Re: Python3 html scraper that supports javascript zljubisic@gmail.com - 2016-05-02 13:11 -0700

csiph-web