Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #35156 > unrolled thread

Q) Why is URLConnection not downloading all the renderable content for me?

Back to article view | Back to comp.lang.java.programmer

Contents

  Q) Why is URLConnection not downloading all the renderable content for me? sandeep6699@yahoo.com - 2016-07-31 09:29 -0700
    Re: Q) Why is URLConnection not downloading all the renderable content for me? Stanimir Stamenkov <s7an10@netscape.net> - 2016-08-02 07:50 +0300
      Re: Q) Why is URLConnection not downloading all the renderable content for me? Stanimir Stamenkov <s7an10@netscape.net> - 2016-08-02 07:53 +0300
      Re: Q) Why is URLConnection not downloading all the renderable content for me? Stanimir Stamenkov <s7an10@netscape.net> - 2016-08-03 09:21 +0300
      Re: Q) Why is URLConnection not downloading all the renderable content for me? Roedy Green <inquiry@mindprod.com> - 2016-08-03 18:38 -0700
    Re: Q) Why is URLConnection not downloading all the renderable content for me? Martin Gregorie <martin@address-in-sig.invalid> - 2016-08-02 10:19 +0000

#35156 — Q) Why is URLConnection not downloading all the renderable content for me?

Hi,

I am using URLConnection to download the contents of the following URL in my Java program:

http://www.pollutionequipmentdirectory.com/category/air?Page=2


My code snippet is as follows:

// ******** Code Snippet Begin ********

try 
{ 
	outputStream = new PrintWriter(new FileOutputStream(outputFileName));

	// try opening the URL
	URLConnection urlConnection = url.openConnection();

	urlStream = url.openStream();

	byte b[] = new byte[1000];
	int numRead = urlStream.read(b);
	String content = new String(b, 0, numRead);
	while ((numRead != -1))
	{
		outputStream.write(content);
		numRead = urlStream.read(b);
		if (numRead != -1)
		{
			content = new String(b, 0, numRead);
			outputStream.write(content);
		}
	}
	urlStream.close();
} 
catch (IOException e) 
{
	System.err.println("ERROR: couldn't open URL " + url);
}
finally
{
	if(outputStream != null) 	outputStream.close();
}

// ******** Code Snippet End ********


However, I am finding that URLConnection does not download the content that I can see if I save the contents of the browser (Chrome in my case).
For example, if I save the browser contents, I see the following content, which I don’t see when I use URLConnection:

// ********Expected content not seen BEGIN ********/
MIL-RAM TECHNOLOGY, Inc. is a leading innovator in industrial gas detection specializing in fixed systems utilizing our unique no false alarms sensor technology. We are a technical, engineering based company with strong emphasis on product design and customization to meet specific application...


Lufft - our philosophy, our services and the reasons why our products are used throughout the world in the fields of industrial climate measurement technology, Meteorology, Renewable Energy, and Intelligent Transportation Systems.


Schutte & Koerting is a designer and manufacturer of equipment for the process and power industries serving the chemical, petroleum, petrochemical, food, pulp and paper, power, metal, refining, water and waste treatment industries among others. Schutte & Koerting products are custom...

// ********Expected content not seen END ********/


Any suggestions would be welcome?

Regards,
Sandeep

[toc] | [next] | [standalone]

#35163

Sun, 31 Jul 2016 09:29:44 -0700 (PDT), sandeep6699@yahoo.com:

> I am using URLConnection to download the contents of the following URL in my Java program:
>
> http://www.pollutionequipmentdirectory.com/category/air?Page=2
>
> My code snippet is as follows:
>
> [...]
> 	URLConnection urlConnection = url.openConnection();
> [...]
>
> However, I am finding that URLConnection does not download the content that I can see if I save the contents of the browser (Chrome in my case).
> [...]
>
> Any suggestions would be welcome?

http://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html#setFollowRedirects-boolean-

     if (urlConnection instanceof HttpURLConnection) {
         HttpURLConnection httpConnection = (HttpURLConnection) 
urlConnection;
         httpConnection.setFollowRedirects(true);
     }

-- 
Stanimir

[toc] | [prev] | [next] | [standalone]

#35164

Tue, 2 Aug 2016 07:50:25 +0300, Stanimir Stamenkov:
> Sun, 31 Jul 2016 09:29:44 -0700 (PDT), sandeep6699@yahoo.com:
>
>> I am using URLConnection to download the contents of the following URL in my Java program:
>>
>> http://www.pollutionequipmentdirectory.com/category/air?Page=2
>>
>> My code snippet is as follows:
>>
>> [...]
>>     URLConnection urlConnection = url.openConnection();
>> [...]
>>
>> However, I am finding that URLConnection does not download the content that I can see if I save the contents of the browser (Chrome in my case).
>> [...]
>>
>> Any suggestions would be welcome?
>
> http://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html#setFollowRedirects-boolean-
>
>
>     if (urlConnection instanceof HttpURLConnection) {
>         HttpURLConnection httpConnection = (HttpURLConnection) urlConnection;
>         httpConnection.setFollowRedirects(true);
>     }

Or just request the URL you're being redirected to:

http://www.pollutionequipmentdirectory.com/category/air

-- 
Stanimir

[toc] | [prev] | [next] | [standalone]

#35181

Tue, 2 Aug 2016 07:50:25 +0300, Stanimir Stamenkov:
> Sun, 31 Jul 2016 09:29:44 -0700 (PDT), sandeep6699@yahoo.com:
>
>> http://www.pollutionequipmentdirectory.com/category/air?Page=2
>>
>> However, I am finding that URLConnection does not download the content that I can see if I save the contents of the browser (Chrome in my case).
>
> http://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html#setFollowRedirects-boolean-
>
>     if (urlConnection instanceof HttpURLConnection) {
>         HttpURLConnection httpConnection = (HttpURLConnection) urlConnection;
>         httpConnection.setFollowRedirects(true);
>     }

Actually this appears the default setting, so the most likely reason 
for your observation has been given by Martin Gregorie in another 
reply to this thread.  The browser additionally fetches related 
resources linked from the content you initially get from the URL 
given.  These related resources may include scripts which the 
browser also evaluates, and they may dynamically change the content 
you're finally seeing in the browser.

Stuff like cookies maintaining specific session in the browser may 
also change the result you're getting.

-- 
Stanimir

[toc] | [prev] | [next] | [standalone]

#35188

On Tue, 2 Aug 2016 07:50:25 +0300, Stanimir Stamenkov
<s7an10@netscape.net> wrote:

>     if (urlConnection instanceof HttpURLConnection) {
>         HttpURLConnection httpConnection = (HttpURLConnection) 
>urlConnection;
>         httpConnection.setFollowRedirects(true);

int estimatedLength = urlc.getContentLength();
final InputStream is = urlc.getInputStream();

The content of the page lives in that InputStream.

[toc] | [prev] | [next] | [standalone]

#35165

On Sun, 31 Jul 2016 09:29:44 -0700, sandeep6699 wrote:

> However, I am finding that URLConnection does not download the content
> that I can see if I save the contents of the browser (Chrome in my
> case).
> For example, if I save the browser contents, I see the following
> content, which I don’t see when I use URLConnection:
>
Your code makes a single request, i.e. it gets just the page identified 
by the URL you opened as a Stream. If you also want material referenced 
by that page (images, content from other linked pages, etc), which seems 
likely since the page you referenced is an index, you need to find all 
the links in the page you've just read and retrieve their content as well.


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web