Groups > comp.lang.java.programmer > #6358

Re: Looking for Java web crawler api

Date	2011-07-21 17:10 -0400
From	Arne Vajhøj <arne@vajhoej.dk>
Newsgroups	comp.lang.java.programmer
Subject	Re: Looking for Java web crawler api
References	<4e1bf464$0$314$14726298@news.sunsite.dk>
Message-ID	<4e2895d1$0$309$14726298@news.sunsite.dk> (permalink)
Organization	SunSITE.dk - Supporting Open source

Show all headers | View raw

On 7/12/2011 3:14 AM, pm wrote:
> Hello, I am working on a project that requires me to do custom search on
> different websites.  I am using Java and while I can write this from
> ground up, I am looking at using existing APIs that can be used due to
> time limit.  So far I have came across Apache's HttpClient.
> 	I am wondering if there are any others that can be effective or
> give more options for web searching/scraping. I plan to create a GUI
> based application and need something quick and effective while not being
> too complex.

http://nutch.apache.org/ should contain a crawler and it comes with
a searchable database (Lucene).

Arne

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Find similar

Thread

Looking for Java web crawler api pm <el_durango@yah00.c0m> - 2011-07-12 07:14 +0000
  Re: Looking for Java web crawler api Bent C Dalager <bcd@pvv.ntnu.no> - 2011-07-12 09:44 +0000
    Re: Looking for Java web crawler api Durango2011 <el_durango@yah00.c0m> - 2011-07-13 06:12 +0000
  Re: Looking for Java web crawler api Roedy Green <see_website@mindprod.com.invalid> - 2011-07-14 12:52 -0700
  Re: Looking for Java web crawler api iadb <freeinternetarticles@gmail.com> - 2011-07-18 16:06 -0700
  Re: Looking for Java web crawler api Durango2011 <el_durango@yah00.c0m> - 2011-07-21 05:49 +0000
  Re: Looking for Java web crawler api Arne Vajhøj <arne@vajhoej.dk> - 2011-07-21 17:10 -0400

csiph-web