Path: csiph.com!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: Stanimir Stamenkov <s7an10@netscape.net>
Newsgroups: comp.lang.java.programmer
Subject: Re: Q) Why is URLConnection not downloading all the renderable content for me?
Date: Tue, 2 Aug 2016 07:53:24 +0300
Organization: A noiseless patient Spider
Lines: 32
Message-ID: <nnp902$rec$2@dont-email.me>
References: <925a87d0-661e-4670-84e4-a9d2b9f9ea89@googlegroups.com> <nnp8qg$rec$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 2 Aug 2016 04:53:23 -0000 (UTC)
Injection-Info: mx02.eternal-september.org; posting-host="c09fc6f411bf9a35d35f3e1e580a61b6"; logging-data="28108"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX18SjJybWH83mmtZBYZeaZVj"
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40
In-Reply-To: <nnp8qg$rec$1@dont-email.me>
X-Face: "@)%Vlap6d%OVYDS}B4YUWE@yUy+^!w/+.q.,c5kjI#+uG?kYP&r/pTjNWgo:g[A,O=AL3/ j&4Le2cau$<e=FPq~GbeZ6DW1Grey[VHn8?ktzvoQtzlg;]&rh8g]CD+_BC7d[gW{Yx&_X*4Q$h&nJ l@t(|4R4:T='&]U#ui+z9OC$.~gJjU93kh:Ojc$Cz0n#I9!~!9ay?E<UlW&5,IFX1*KtA>MYEFst$y y"m]M7tq\S(>[a$x0x8%S
Cancel-Lock: sha1:cV5Nj1x19+bxCLBRuLTcJUEdsoc=
Xref: csiph.com comp.lang.java.programmer:35164

Tue, 2 Aug 2016 07:50:25 +0300, Stanimir Stamenkov:
> Sun, 31 Jul 2016 09:29:44 -0700 (PDT), sandeep6699@yahoo.com:
>
>> I am using URLConnection to download the contents of the following URL in my Java program:
>>
>> http://www.pollutionequipmentdirectory.com/category/air?Page=2
>>
>> My code snippet is as follows:
>>
>> [...]
>>     URLConnection urlConnection = url.openConnection();
>> [...]
>>
>> However, I am finding that URLConnection does not download the content that I can see if I save the contents of the browser (Chrome in my case).
>> [...]
>>
>> Any suggestions would be welcome?
>
> http://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html#setFollowRedirects-boolean-
>
>
>     if (urlConnection instanceof HttpURLConnection) {
>         HttpURLConnection httpConnection = (HttpURLConnection) urlConnection;
>         httpConnection.setFollowRedirects(true);
>     }

Or just request the URL you're being redirected to:

http://www.pollutionequipmentdirectory.com/category/air

-- 
Stanimir