Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #31884 > unrolled thread

how to capture full webpage in java

Started byNavneet Mathpal <navneetmathpal93@gmail.com>
First post2015-06-30 02:13 -0700
Last post2015-07-02 12:58 -0700
Articles 5 — 3 participants

Back to article view | Back to comp.lang.java.programmer

Contents

  how to capture full webpage in java Navneet Mathpal <navneetmathpal93@gmail.com> - 2015-06-30 02:13 -0700
    Re: how to capture full webpage in java e.d.programmer@gmail.com - 2015-06-30 05:34 -0700
    Re: how to capture full webpage in java Eternal Hope <laughingspam99@gmail.com> - 2015-07-02 20:46 +0100
      Re: how to capture full webpage in java Eternal Hope <laughingspam99@gmail.com> - 2015-07-02 20:50 +0100
        Re: how to capture full webpage in java e.d.programmer@gmail.com - 2015-07-02 12:58 -0700

#31884 — how to capture full webpage in java

FromNavneet Mathpal <navneetmathpal93@gmail.com>
Date2015-06-30 02:13 -0700
Subjecthow to capture full webpage in java
Message-ID<b93849bb-23f7-4867-bd3e-81ba9157b5c6@googlegroups.com>
Hi ,
Have you tried to capture the full webpage using java ?
I want to capture full webpage using java (ex: http://stackoverflow.com) . I am able to capture the first screen but can not scoll down till the web-page ends ?

I tried to do it with Selenium , but could not do it(some jar problem occuring all the time )


could you please help. ?

[toc] | [next] | [standalone]

#31885

Frome.d.programmer@gmail.com
Date2015-06-30 05:34 -0700
Message-ID<fee8fba7-400a-47a1-ae74-9597fb1b5ff9@googlegroups.com>
In reply to#31884
On Tuesday, June 30, 2015 at 5:13:42 AM UTC-4, Navneet Mathpal wrote:
> Hi ,
> Have you tried to capture the full webpage using java ?
> I want to capture full webpage using java (ex: http://stackoverflow.com) . I am able to capture the first screen but can not scoll down till the web-page ends ?
> 
> I tried to do it with Selenium , but could not do it(some jar problem occuring all the time )
> 
> 
> could you please help. ?

Do you have some sample code with what you say you've tried so far?

[toc] | [prev] | [next] | [standalone]

#31927

FromEternal Hope <laughingspam99@gmail.com>
Date2015-07-02 20:46 +0100
Message-ID<mn44cq$o4i$1@dont-email.me>
In reply to#31884
On 30/06/2015 10:13, Navneet Mathpal wrote:
> Hi ,
> Have you tried to capture the full webpage using java ?
> I want to capture full webpage using java (ex: http://stackoverflow.com) . I am able to capture the first screen but can not scoll down till the web-page ends ?
>
> I tried to do it with Selenium , but could not do it(some jar problem occuring all the time )
>
>
> could you please help. ?

package usenet.foo;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

public class PageReader {
	
	public static void main(String[] args){
		URL url = null;
	    BufferedReader reader = null;
	    StringBuilder stringBuilder;
	
	    try{
	
	      url = new URL("http://stackoverflow.com");
	      HttpURLConnection connection = (HttpURLConnection) 
url.openConnection();
	      connection.setRequestMethod("GET");
	      connection.setReadTimeout(15*1000);
	      connection.connect();
	      reader = new BufferedReader(new 
InputStreamReader(connection.getInputStream()));
	      stringBuilder = new StringBuilder();
	
	      String line = null;
	      while ((line = reader.readLine()) != null) {
	        stringBuilder.append(line + "\n");
	      }
	      //parse or whatever
	      System.out.println(stringBuilder.toString());
	    }
	    catch (Exception e){
	      e.printStackTrace();
	  }
	}
}

[toc] | [prev] | [next] | [standalone]

#31928

FromEternal Hope <laughingspam99@gmail.com>
Date2015-07-02 20:50 +0100
Message-ID<mn44k2$o4i$2@dont-email.me>
In reply to#31927
On 02/07/2015 20:46, Eternal Hope wrote:
> On 30/06/2015 10:13, Navneet Mathpal wrote:
>> Hi ,
>> Have you tried to capture the full webpage using java ?
>> I want to capture full webpage using java (ex:
>> http://stackoverflow.com) . I am able to capture the first screen but
>> can not scoll down till the web-page ends ?
>>
>> I tried to do it with Selenium , but could not do it(some jar problem
>> occuring all the time )
>>
>>
>> could you please help. ?
>
> package usenet.foo;
>
> import java.io.BufferedReader;
> import java.io.InputStreamReader;
> import java.net.HttpURLConnection;
> import java.net.URL;
>
> public class PageReader {
>
>      public static void main(String[] args){
>          URL url = null;
>          BufferedReader reader = null;
>          StringBuilder stringBuilder;
>
>          try{
>
>            url = new URL("http://stackoverflow.com");
>            HttpURLConnection connection = (HttpURLConnection)
> url.openConnection();
>            connection.setRequestMethod("GET");
>            connection.setReadTimeout(15*1000);
>            connection.connect();
>            reader = new BufferedReader(new
> InputStreamReader(connection.getInputStream()));
>            stringBuilder = new StringBuilder();
>
>            String line = null;
>            while ((line = reader.readLine()) != null) {
>              stringBuilder.append(line + "\n");
>            }
>            //parse or whatever
>            System.out.println(stringBuilder.toString());
>          }
>          catch (Exception e){
>            e.printStackTrace();
>        }
>      }
> }

That took 2 minutes to find, 2 minutes to modify and 30 seconds to test :-)

[toc] | [prev] | [next] | [standalone]

#31930

Frome.d.programmer@gmail.com
Date2015-07-02 12:58 -0700
Message-ID<3a6ee5dc-b09f-4fc3-b9b0-95e73f303d5b@googlegroups.com>
In reply to#31928
On Thursday, July 2, 2015 at 3:50:54 PM UTC-4, Eternal Hope wrote:
> On 02/07/2015 20:46, Eternal Hope wrote:
> > On 30/06/2015 10:13, Navneet Mathpal wrote:
> >> Hi ,
> >> Have you tried to capture the full webpage using java ?
> >> I want to capture full webpage using java (ex:
> >> http://stackoverflow.com) . I am able to capture the first screen but
> >> can not scoll down till the web-page ends ?
> >>
> >> I tried to do it with Selenium , but could not do it(some jar problem
> >> occuring all the time )
> >>
> >>
> >> could you please help. ?
> >
> > package usenet.foo;
> >
> > import java.io.BufferedReader;
> > import java.io.InputStreamReader;
> > import java.net.HttpURLConnection;
> > import java.net.URL;
> >
> > public class PageReader {
> >
> >      public static void main(String[] args){
> >          URL url = null;
> >          BufferedReader reader = null;
> >          StringBuilder stringBuilder;
> >
> >          try{
> >
> >            url = new URL("http://stackoverflow.com");
> >            HttpURLConnection connection = (HttpURLConnection)
> > url.openConnection();
> >            connection.setRequestMethod("GET");
> >            connection.setReadTimeout(15*1000);
> >            connection.connect();
> >            reader = new BufferedReader(new
> > InputStreamReader(connection.getInputStream()));
> >            stringBuilder = new StringBuilder();
> >
> >            String line = null;
> >            while ((line = reader.readLine()) != null) {
> >              stringBuilder.append(line + "\n");
> >            }
> >            //parse or whatever
> >            System.out.println(stringBuilder.toString());
> >          }
> >          catch (Exception e){
> >            e.printStackTrace();
> >        }
> >      }
> > }
> 
> That took 2 minutes to find, 2 minutes to modify and 30 seconds to test :-)

Web is a funny thing.  My guess was the solution would be similar to my solution, where I wrote a drag and drop handler to drag an image from a web page and drop it into a JComponent.  Initially I only got part of a file.  I had to use a different method to retrieve the entire file.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web