Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #9505 > unrolled thread

Problem in Web scraping

Back to article view | Back to comp.lang.javascript

Contents

  Problem in Web scraping karthiga <cutekarthi2989@gmail.com> - 2011-12-21 21:24 -0800
    Re: Problem in Web scraping Bart Van der Donck <bart@nijlen.com> - 2011-12-22 02:42 -0800
    Re: Problem in Web scraping Scott Sauyet <scott.sauyet@gmail.com> - 2011-12-22 06:07 -0800
      Re: Problem in Web scraping Jeremy J Starcher <r3jjs@yahoo.com> - 2012-01-11 17:13 +0000

#9505 — Problem in Web scraping

Is it possible to fetch dynamic content from two websites at the same
page.

eg: www.imdb.com from this site we can search the movies, fetch the
rating,content,director and from www.watchfreemovies.ch we can search
the same film name, fetch the download links, and versions

[toc] | [next] | [standalone]

#9508

karthiga wrote:

> Is it possible to fetch dynamic content from two websites at the same
> page.
> eg:www.imdb.comfrom this site we can search the movies, fetch the
> rating,content,director and fromwww.watchfreemovies.chwe can search
> the same film name, fetch the download links, and versions

Yes, this can be achieved by so-called content-grabbers (see e.g. PHP,
Perl, ...). I once wrote www.ajax-cross-domain.com which offers a
javascript solution, although the core mechanism is (must be) server-
side.

The real problem of your plan is that you're never sure that it will
continue to work. At any moment, the websites in question can alter
their HTML, potentially causing your script to fail every time. Large
websites such as IMDB usually tend to modify their pages quite often.

Legal issues are also involved. Unless explicit approval, you are
usually not allowed to systematically parse their data for your own
purposes.

Hope this helps,

--
 Bart

[toc] | [prev] | [next] | [standalone]

#9509

karthiga wrote:
> Is it possible to fetch dynamic content from two websites at the same
> page.
>
> eg: www.imdb.comfrom this site we can search the movies, fetch the
> rating,content,director and from www.watchfreemovies.ch we can search
> the same film name, fetch the download links, and versions

In the browser this is generally not possible [1], unless the sites
cooperate by, for example, providing a JSONP API [2] or by supporting
CORS [3].  But it's very easy to write a server-side proxy that loads
the external sites as though they were local.

  -- Scott

[1] http://en.wikipedia.org/wiki/Same_origin_policy
[2] http://en.wikipedia.org/wiki/JSONP
[3] http://en.wikipedia.org/wiki/Cross-Origin_Resource_Sharing

[toc] | [prev] | [next] | [standalone]

#10136

On Thu, 22 Dec 2011 06:07:30 -0800, Scott Sauyet wrote:

> karthiga wrote:
>> Is it possible to fetch dynamic content from two websites at the same
>> page.
> In the browser this is generally not possible [1], unless the sites
> cooperate by, for example, providing a JSONP API [2] or by supporting
> CORS [3].  But it's very easy to write a server-side proxy that loads
> the external sites as though they were local.

If this is for your own personal use (and not something you want to share/
distribute) then a UserScript[1] isn't limited by the same domain origin 
limitation.

[1] I have a lot of experience doing similar things in IE and Firefox, 
both using a plugin to get Userscript abilities.  Some other browsers 
(like Chrome) have some UserScript abilities built in, but YMMV.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.javascript

csiph-web