Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.misc > #26375 > unrolled thread

[LINK] Google begins requiring JavaScript for Google Search

Back to article view | Back to comp.misc

Contents

  [LINK] Google begins requiring JavaScript for Google Search not@telling.you.invalid (Computer Nerd Kev) - 2025-01-21 07:10 +1000
    Re: [LINK] Google begins requiring JavaScript for Google Search D <nospam@example.net> - 2025-01-21 10:23 +0100
    Re: [LINK] Google begins requiring JavaScript for Google Search Sylvia Else <sylvia@email.invalid> - 2025-01-21 20:18 +0800
      Re: [LINK] Google begins requiring JavaScript for Google Search Jan van den Broek <balglaas@dds.nl> - 2025-01-21 12:56 +0000
        Re: [LINK] Google begins requiring JavaScript for Google Search D <nospam@example.net> - 2025-01-21 19:18 +0100
      Re: [LINK] Google begins requiring JavaScript for Google Search Salvador Mirzo <smirzo@example.com> - 2025-01-21 10:33 -0300
        Re: [LINK] Google begins requiring JavaScript for Google Search Rich <rich@example.invalid> - 2025-01-21 18:12 +0000
        Re: [LINK] Google begins requiring JavaScript for Google Search not@telling.you.invalid (Computer Nerd Kev) - 2025-01-22 06:56 +1000
          Re: [LINK] Google begins requiring JavaScript for Google Search Ivan Shmakov <ivan@siamics.netREMOVE.invalid> - 2025-01-23 19:33 +0000
            Re: [LINK] Google begins requiring JavaScript for Google Search Sylvia Else <sylvia@email.invalid> - 2025-01-24 13:30 +0800
              Re: [LINK] Google begins requiring JavaScript for Google Search Richmond <dnomhcir@gmx.com> - 2025-01-27 20:56 +0000
                Re: [LINK] Google begins requiring JavaScript for Google Search Richmond <dnomhcir@gmx.com> - 2025-01-27 21:02 +0000
        Re: [LINK] Google begins requiring JavaScript for Google Search candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> - 2025-01-29 19:00 +0000
          Re: [LINK] Google begins requiring JavaScript for Google Search Rich <rich@example.invalid> - 2025-01-29 19:33 +0000
            Re: [LINK] Google begins requiring JavaScript for Google Search Richmond <dnomhcir@gmx.com> - 2025-01-29 20:04 +0000
              Re: [LINK] Google begins requiring JavaScript for Google Search kludge@panix.com (Scott Dorsey) - 2025-01-29 22:15 +0000
                Re: [LINK] Google begins requiring JavaScript for Google Search D <nospam@example.net> - 2025-01-30 10:50 +0100
                Re: [LINK] Google begins requiring JavaScript for Google Search Ivan Shmakov <ivan@siamics.netREMOVE.invalid> - 2025-01-30 18:47 +0000
              Re: [LINK] Google begins requiring JavaScript for Google Search Rich <rich@example.invalid> - 2025-01-30 00:53 +0000
                Re: [LINK] Google begins requiring JavaScript for Google Search yeti <yeti@tilde.institute> - 2025-01-30 01:53 +0042
                  Re: [LINK] Google begins requiring JavaScript for Google Search Rich <rich@example.invalid> - 2025-01-30 03:38 +0000
                    Re: [LINK] Google begins requiring JavaScript for Google Search yeti <yeti@tilde.institute> - 2025-01-30 04:43 +0042
          Re: [LINK] Google begins requiring JavaScript for Google Search Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-02-24 05:38 +0000
    Re: [LINK] Google begins requiring JavaScript for Google Search Jim Jackson <jj@franjam.org.uk> - 2025-01-21 15:30 +0000
      Re: [LINK] Google begins requiring JavaScript for Google Search D <noreply@mixmin.net> - 2025-01-21 16:26 +0000
      Re: [LINK] Google begins requiring JavaScript for Google Search D <nospam@example.net> - 2025-01-21 19:21 +0100
    Re: [LINK] Google begins requiring JavaScript for Google Search Andy Burns <usenet@andyburns.uk> - 2025-01-21 19:54 +0000
      Re: [LINK] Google begins requiring JavaScript for Google Search not@telling.you.invalid (Computer Nerd Kev) - 2025-01-22 06:47 +1000
        Re: [LINK] Google begins requiring JavaScript for Google Search yeti <yeti@tilde.institute> - 2025-01-21 22:49 +0042
          Re: [LINK] Google begins requiring JavaScript for Google Search Computer Nerd Kev <not@telling.you.invalid> - 2025-01-22 13:04 +1000
            Re: [LINK] Google begins requiring JavaScript for Google Search yeti <yeti@tilde.institute> - 2025-01-22 12:58 +0042
            Re: [LINK] Google begins requiring JavaScript for Google Search kludge@panix.com (Scott Dorsey) - 2025-01-25 15:15 +0000
    Re: [LINK] Google begins requiring JavaScript for Google Search Jerry Peters <jerry@example.invalid> - 2025-01-21 20:57 +0000

Page 1 of 2  [1] 2  Next page →

#26375 — [LINK] Google begins requiring JavaScript for Google Search

Google begins requiring JavaScript for Google Search
 by Thom Holwerda 2025-01-18
 - https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/

" Google says it has begun requiring users to turn on JavaScript, the 
  widely used programming language to make web pages interactive, in 
  order to use Google Search.
  In an email to TechCrunch, a company spokesperson claimed that the 
  change is intended to "better protect" Google Search against 
  malicious activity, such as bots and spam, and to improve the 
  overall Google Search experience for users. The spokesperson noted 
  that, without JavaScript, many Google Search features won't work 
  properly and that the quality of search results tends to be 
  degraded.
 Kyle Wiggers at TechCrunch
 
 One of the strangely odd compliments you could give Google Search 
 is that it would load even on the weirdest or oldest browsers, 
 simply because it didn't require JavaScript. Whether I loaded 
 Google Search in the JS-less Dillo, Blazer on PalmOS, or the latest 
 Firefox, I'd end up with a search box I could type something into 
 and search. Sure, beyond that the web would be, shall we say, 
 problematic, but at least Google Search worked. With this move, 
 Google will end such compatibility, which was most likely a side 
 effect more than policy." ...

I switched from Google to Duck Duck Go (Lite) many years ago, but
it's annoying that I'll have to find another search engine to use
as a fall-back for when DDG breaks, since I do most of my Web
browsing in Dillo.

-- 
__          __
#_ < |\| |< _#

[toc] | [next] | [standalone]

#26377

On Mon, 21 Jan 2025, Computer Nerd Kev wrote:

> Google begins requiring JavaScript for Google Search
> by Thom Holwerda 2025-01-18
> - https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/
>
> " Google says it has begun requiring users to turn on JavaScript, the
>  widely used programming language to make web pages interactive, in
>  order to use Google Search.
>  In an email to TechCrunch, a company spokesperson claimed that the
>  change is intended to "better protect" Google Search against
>  malicious activity, such as bots and spam, and to improve the
>  overall Google Search experience for users. The spokesperson noted
>  that, without JavaScript, many Google Search features won't work
>  properly and that the quality of search results tends to be
>  degraded.
> Kyle Wiggers at TechCrunch
>
> One of the strangely odd compliments you could give Google Search
> is that it would load even on the weirdest or oldest browsers,
> simply because it didn't require JavaScript. Whether I loaded
> Google Search in the JS-less Dillo, Blazer on PalmOS, or the latest
> Firefox, I'd end up with a search box I could type something into
> and search. Sure, beyond that the web would be, shall we say,
> problematic, but at least Google Search worked. With this move,
> Google will end such compatibility, which was most likely a side
> effect more than policy." ...
>
> I switched from Google to Duck Duck Go (Lite) many years ago, but
> it's annoying that I'll have to find another search engine to use
> as a fall-back for when DDG breaks, since I do most of my Web
> browsing in Dillo.

Can't you use the !g on ddg? Maybe ddg sanitizes the google output a bit? 
If that doesn't work, I recommend startpage.com which is just an interface 
to google.

[toc] | [prev] | [next] | [standalone]

#26378

On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
> Google begins requiring JavaScript for Google Search
>   by Thom Holwerda 2025-01-18
>   - https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/
> 
> " Google says it has begun requiring users to turn on JavaScript, the
>    widely used programming language to make web pages interactive, in
>    order to use Google Search.
>    In an email to TechCrunch, a company spokesperson claimed that the
>    change is intended to "better protect" Google Search against
>    malicious activity, such as bots and spam, and to improve the
>    overall Google Search experience for users. The spokesperson noted
>    that, without JavaScript, many Google Search features won't work
>    properly and that the quality of search results tends to be
>    degraded.
>   Kyle Wiggers at TechCrunch
>   
>   One of the strangely odd compliments you could give Google Search
>   is that it would load even on the weirdest or oldest browsers,
>   simply because it didn't require JavaScript. Whether I loaded
>   Google Search in the JS-less Dillo, Blazer on PalmOS, or the latest
>   Firefox, I'd end up with a search box I could type something into
>   and search. Sure, beyond that the web would be, shall we say,
>   problematic, but at least Google Search worked. With this move,
>   Google will end such compatibility, which was most likely a side
>   effect more than policy." ...
> 
> I switched from Google to Duck Duck Go (Lite) many years ago, but
> it's annoying that I'll have to find another search engine to use
> as a fall-back for when DDG breaks, since I do most of my Web
> browsing in Dillo.
> 

How is this going to '"better protect" Google Search against malicious 
activity, such as bots and spam'?

Sylvia.

[toc] | [prev] | [next] | [standalone]

#26379

2025-01-21, Sylvia Else <sylvia@email.invalid> schrieb:

[Schnipp]

> How is this going to '"better protect" Google Search against malicious 
> activity, such as bots and spam'?
>
> Sylvia.

Simple, it won't, but it sounds nice.

-- 
Jan van den Broek
balglaas@dds.nl                                              0xAFDAD00D
                                        http://huizen.dds.nl/~balglaas/

[toc] | [prev] | [next] | [standalone]

#26384

On Tue, 21 Jan 2025, Jan van den Broek wrote:

> 2025-01-21, Sylvia Else <sylvia@email.invalid> schrieb:
>
> [Schnipp]
>
>> How is this going to '"better protect" Google Search against malicious
>> activity, such as bots and spam'?
>>
>> Sylvia.
>
> Simple, it won't, but it sounds nice.

This is the truth!

[toc] | [prev] | [next] | [standalone]

#26380

Sylvia Else <sylvia@email.invalid> writes:

> On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
>> Google begins requiring JavaScript for Google Search
>>   by Thom Holwerda 2025-01-18
>>   - https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/
>> " Google says it has begun requiring users to turn on JavaScript,
>> the
>>    widely used programming language to make web pages interactive, in
>>    order to use Google Search.
>>    In an email to TechCrunch, a company spokesperson claimed that the
>>    change is intended to "better protect" Google Search against
>>    malicious activity, such as bots and spam, and to improve the
>>    overall Google Search experience for users.

[...]

> How is this going to '"better protect" Google Search against malicious
> activity, such as bots and spam'?

I believe the idea is that if the robot doesn't speak Javascript, it's
an easy denial by the web server.  And making bots speak Javascript is
one step up.  And with Javascript they can likely monitor things like
mouse movement to detect whether the user is a human or a robot.  

I'm not approving the idea; just sharing what I think they might have in
mind when they say Javascript will help them fend off robots.

[toc] | [prev] | [next] | [standalone]

#26383

Salvador Mirzo <smirzo@example.com> wrote:
> Sylvia Else <sylvia@email.invalid> writes:
> 
>> On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
>>> Google begins requiring JavaScript for Google Search
>>>   by Thom Holwerda 2025-01-18
>>>   - 
>>> https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/ 
>>> " Google says it has begun requiring users to turn on JavaScript, 
>>>   the widely used programming language to make web pages 
>>>   interactive, in order to use Google Search.  In an email to 
>>>   TechCrunch, a company spokesperson claimed that the change is 
>>>   intended to "better protect" Google Search against malicious 
>>>   activity, such as bots and spam, and to improve the overall 
>>>   Google Search experience for users.
> 
> [...]
> 
>> How is this going to '"better protect" Google Search against 
>> malicious activity, such as bots and spam'?
> 
> I believe the idea is that if the robot doesn't speak Javascript, 
> it's an easy denial by the web server.  And making bots speak 
> Javascript is one step up.  And with Javascript they can likely 
> monitor things like mouse movement to detect whether the user is a 
> human or a robot.
> 
> I'm not approving the idea; just sharing what I think they might have 
> in mind when they say Javascript will help them fend off robots.

Yes, this is probably the 'excuse' they would offer up if pressed (for 
bots -- for SPAM, no idea).

But the part they forget is that the reason they have such a 'bots' 
problem is the revenue the bot authors can obtain by gaming google 
search.  All requiring JS will do is result in those same scammers 
"building a better bot" -- i.e., the revenue stream is enough they will 
put in the effort to make their bots speak JS, and google will be back 
where they started.

The *real* reason, which they will likely never admit to, is likely 
that the advertising overlords in control of what is left of the old 
"don't be evil" google figured out they can gain more "data" on users 
by requiring JS than not, and so the change is solely to hoover up more 
data and gain more ad dollars for the mothership.

[toc] | [prev] | [next] | [standalone]

#26388

Salvador Mirzo <smirzo@example.com> wrote:
> Sylvia Else <sylvia@email.invalid> writes:
>> How is this going to '"better protect" Google Search against malicious
>> activity, such as bots and spam'?
> 
> I believe the idea is that if the robot doesn't speak Javascript, it's
> an easy denial by the web server.  And making bots speak Javascript is
> one step up.  And with Javascript they can likely monitor things like
> mouse movement to detect whether the user is a human or a robot.  

Which of course is one of Google's main businesses, with their
Captchas that don't always need to show a puzzle in order to
validate users as humans. So if anyone _thinks_ they can achieve
that, you'd expect it to be Google.

-- 
__          __
#_ < |\| |< _#

[toc] | [prev] | [next] | [standalone]

#26394

>>>>> On 2025-01-21, Computer Nerd Kev wrote:
>>>>> Salvador Mirzo <smirzo@example.com> wrote:
>>>>> Sylvia Else <sylvia@email.invalid> writes:

 >>> How is this going to '"better protect" Google Search against
 >>> malicious activity, such as bots and spam'?

 >> I believe the idea is that if the robot doesn't speak Javascript,
 >> it's an easy denial by the web server.  And making bots speak
 >> Javascript is one step up.  And with Javascript they can likely
 >> monitor things like mouse movement to detect whether the user
 >> is a human or a robot.

 > Which of course is one of Google's main businesses, with their
 > Captchas that don't always need to show a puzzle in order to
 > validate users as humans.  So if anyone _thinks_ they can achieve
 > that, you'd expect it to be Google.

	And they don't even need it to be perfect: a robot that
	implements the relevant browser APIs, while possible, /will/
	be costlier to run and maintain, thus reducing the profits of
	the robot operators, in turn disincentivizing them.

	Even if that doesn't solve the problem altogether, it will
	still likely result in less load for their servers.

	Not that it invalidates any other reasons they might want to
	require Javascript / APIs regardless, mind you.

[toc] | [prev] | [next] | [standalone]

#26399

On 24-Jan-25 3:33 am, Ivan Shmakov wrote:
>>>>>> On 2025-01-21, Computer Nerd Kev wrote:
>>>>>> Salvador Mirzo <smirzo@example.com> wrote:
>>>>>> Sylvia Else <sylvia@email.invalid> writes:
> 
>   >>> How is this going to '"better protect" Google Search against
>   >>> malicious activity, such as bots and spam'?
> 
>   >> I believe the idea is that if the robot doesn't speak Javascript,
>   >> it's an easy denial by the web server.  And making bots speak
>   >> Javascript is one step up.  And with Javascript they can likely
>   >> monitor things like mouse movement to detect whether the user
>   >> is a human or a robot.
> 
>   > Which of course is one of Google's main businesses, with their
>   > Captchas that don't always need to show a puzzle in order to
>   > validate users as humans.  So if anyone _thinks_ they can achieve
>   > that, you'd expect it to be Google.
> 
> 	And they don't even need it to be perfect: a robot that
> 	implements the relevant browser APIs, while possible, /will/
> 	be costlier to run and maintain, thus reducing the profits of
> 	the robot operators, in turn disincentivizing them.
> 
> 	Even if that doesn't solve the problem altogether, it will
> 	still likely result in less load for their servers.
> 
> 	Not that it invalidates any other reasons they might want to
> 	require Javascript / APIs regardless, mind you.

A bot only needs to be able to send the correct data to the server. how 
difficult that is obviously depends on the details of the Javascript's 
interactions with the server, but frequent interactions themselves 
create a higher server load.

One example would be the mouse-movement based human detection. If the 
script just sends a yes/no message to the server, then the bot doesn't 
need to try to emulate a human at all.

Sylvia.

[toc] | [prev] | [next] | [standalone]

#26407

Sylvia Else <sylvia@email.invalid> writes:

> On 24-Jan-25 3:33 am, Ivan Shmakov wrote:
>>>>>>> On 2025-01-21, Computer Nerd Kev wrote:
>>>>>>> Salvador Mirzo <smirzo@example.com> wrote:
>>>>>>> Sylvia Else <sylvia@email.invalid> writes:
>>   >>> How is this going to '"better protect" Google Search against
>>   >>> malicious activity, such as bots and spam'?
>>   >> I believe the idea is that if the robot doesn't speak
>> Javascript,
>>   >> it's an easy denial by the web server.  And making bots speak
>>   >> Javascript is one step up.  And with Javascript they can likely
>>   >> monitor things like mouse movement to detect whether the user
>>   >> is a human or a robot.
>>   > Which of course is one of Google's main businesses, with their
>>   > Captchas that don't always need to show a puzzle in order to
>>   > validate users as humans.  So if anyone _thinks_ they can achieve
>>   > that, you'd expect it to be Google.
>> 	And they don't even need it to be perfect: a robot that
>> 	implements the relevant browser APIs, while possible, /will/
>> 	be costlier to run and maintain, thus reducing the profits of
>> 	the robot operators, in turn disincentivizing them.
>> 	Even if that doesn't solve the problem altogether, it will
>> 	still likely result in less load for their servers.
>> 	Not that it invalidates any other reasons they might want to
>> 	require Javascript / APIs regardless, mind you.
>
> A bot only needs to be able to send the correct data to the
> server. how difficult that is obviously depends on the details of the
> Javascript's interactions with the server, but frequent interactions
> themselves create a higher server load.
>
> One example would be the mouse-movement based human detection. If the
> script just sends a yes/no message to the server, then the bot doesn't
> need to try to emulate a human at all.
>
> Sylvia.

That's useful. I set my Seamonkey user agent string to a Lynx user agent
string and now google search works without javascript.

[toc] | [prev] | [next] | [standalone]

#26408

Richmond <dnomhcir@gmx.com> writes:

> Sylvia Else <sylvia@email.invalid> writes:
>
>> On 24-Jan-25 3:33 am, Ivan Shmakov wrote:
>>>>>>>> On 2025-01-21, Computer Nerd Kev wrote:
>>>>>>>> Salvador Mirzo <smirzo@example.com> wrote:
>>>>>>>> Sylvia Else <sylvia@email.invalid> writes:
>>>   >>> How is this going to '"better protect" Google Search against
>>>   >>> malicious activity, such as bots and spam'?
>>>   >> I believe the idea is that if the robot doesn't speak
>>> Javascript,
>>>   >> it's an easy denial by the web server.  And making bots speak
>>>   >> Javascript is one step up.  And with Javascript they can likely
>>>   >> monitor things like mouse movement to detect whether the user
>>>   >> is a human or a robot.
>>>   > Which of course is one of Google's main businesses, with their
>>>   > Captchas that don't always need to show a puzzle in order to
>>>   > validate users as humans.  So if anyone _thinks_ they can achieve
>>>   > that, you'd expect it to be Google.
>>> 	And they don't even need it to be perfect: a robot that
>>> 	implements the relevant browser APIs, while possible, /will/
>>> 	be costlier to run and maintain, thus reducing the profits of
>>> 	the robot operators, in turn disincentivizing them.
>>> 	Even if that doesn't solve the problem altogether, it will
>>> 	still likely result in less load for their servers.
>>> 	Not that it invalidates any other reasons they might want to
>>> 	require Javascript / APIs regardless, mind you.
>>
>> A bot only needs to be able to send the correct data to the
>> server. how difficult that is obviously depends on the details of the
>> Javascript's interactions with the server, but frequent interactions
>> themselves create a higher server load.
>>
>> One example would be the mouse-movement based human detection. If the
>> script just sends a yes/no message to the server, then the bot doesn't
>> need to try to emulate a human at all.
>>
>> Sylvia.
>
> That's useful. I set my Seamonkey user agent string to a Lynx user agent
> string and now google search works without javascript.

Sorry, I replied to the wrong article.

[toc] | [prev] | [next] | [standalone]

#26413

Salvador Mirzo <smirzo@example.com> wrote at 13:33 this Tuesday (GMT):
> Sylvia Else <sylvia@email.invalid> writes:
>
>> On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
>>> Google begins requiring JavaScript for Google Search
>>>   by Thom Holwerda 2025-01-18
>>>   - https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/
>>> " Google says it has begun requiring users to turn on JavaScript,
>>> the
>>>    widely used programming language to make web pages interactive, in
>>>    order to use Google Search.
>>>    In an email to TechCrunch, a company spokesperson claimed that the
>>>    change is intended to "better protect" Google Search against
>>>    malicious activity, such as bots and spam, and to improve the
>>>    overall Google Search experience for users.
>
> [...]
>
>> How is this going to '"better protect" Google Search against malicious
>> activity, such as bots and spam'?
>
> I believe the idea is that if the robot doesn't speak Javascript, it's
> an easy denial by the web server.  And making bots speak Javascript is
> one step up.  And with Javascript they can likely monitor things like
> mouse movement to detect whether the user is a human or a robot.  
>
> I'm not approving the idea; just sharing what I think they might have in
> mind when they say Javascript will help them fend off robots.


It would also make it harder to scrape, since I /think/ web scrapers
don't run JS by default.
-- 
user <candycane> is generated from /dev/urandom

[toc] | [prev] | [next] | [standalone]

#26414

candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> wrote:
> Salvador Mirzo <smirzo@example.com> wrote at 13:33 this Tuesday (GMT):
>> Sylvia Else <sylvia@email.invalid> writes:
>>
>>> On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
>>>> Google begins requiring JavaScript for Google Search
>>>>   by Thom Holwerda 2025-01-18 - 
>>>>   https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/ 
>>>>   " Google says it has begun requiring users to turn on 
>>>>   JavaScript, the widely used programming language to make web 
>>>>   pages interactive, in order to use Google Search.  In an email 
>>>>   to TechCrunch, a company spokesperson claimed that the change is 
>>>>   intended to "better protect" Google Search against malicious 
>>>>   activity, such as bots and spam, and to improve the overall 
>>>>   Google Search experience for users.
>>
>> [...]
>>
>>> How is this going to '"better protect" Google Search against 
>>> malicious activity, such as bots and spam'?
>>
>> I believe the idea is that if the robot doesn't speak Javascript, 
>> it's an easy denial by the web server.  And making bots speak 
>> Javascript is one step up.  And with Javascript they can likely 
>> monitor things like mouse movement to detect whether the user is a 
>> human or a robot.
>>
>> I'm not approving the idea; just sharing what I think they might 
>> have in mind when they say Javascript will help them fend off 
>> robots.
> 
> It would also make it harder to scrape, since I /think/ web scrapers 
> don't run JS by default.

Which just means this will push web scrapers to start running JS.

They don't run JS (yet) because they have not needed to run JS to do 
their scraping.  But if JS is required, and they want to scrape bad 
enough, they will put in support for running JS.

[toc] | [prev] | [next] | [standalone]

#26415

Rich <rich@example.invalid> writes:

> candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> wrote:
>> Salvador Mirzo <smirzo@example.com> wrote at 13:33 this Tuesday
>>(GMT): > Sylvia Else <sylvia@email.invalid> writes:
>>>
>>>> On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
>>>>> Google begins requiring JavaScript for Google Search by Thom
>>>>>   Holwerda 2025-01-18 -
>>>>>   https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/
>>>>>   " Google says it has begun requiring users to turn on
>>>>>   JavaScript, the widely used programming language to make web
>>>>>   pages interactive, in order to use Google Search.  In an email
>>>>>   to TechCrunch, a company spokesperson claimed that the change is
>>>>>   intended to "better protect" Google Search against malicious
>>>>>   activity, such as bots and spam, and to improve the overall
>>>>>   Google Search experience for users.
>>>
>>> [...]
>>>
>>>> How is this going to '"better protect" Google Search against
>>>> malicious activity, such as bots and spam'?
>>>
>>> I believe the idea is that if the robot doesn't speak Javascript,
>>> it's an easy denial by the web server.  And making bots speak
>>> Javascript is one step up.  And with Javascript they can likely
>>> monitor things like mouse movement to detect whether the user is a
>>> human or a robot.
>>>
>>> I'm not approving the idea; just sharing what I think they might
>>> have in mind when they say Javascript will help them fend off
>>> robots.
>> 
>> It would also make it harder to scrape, since I /think/ web scrapers
>> don't run JS by default.
>
> Which just means this will push web scrapers to start running JS.
>
> They don't run JS (yet) because they have not needed to run JS to do
> their scraping.  But if JS is required, and they want to scrape bad
> enough, they will put in support for running JS.

Why can't web scrapers just pretend to be Lynx browsers?

[toc] | [prev] | [next] | [standalone]

#26417

In article <861pwl5qu8.fsf@example.com>, Richmond  <dnomhcir@gmx.com> wrote:
>
>Why can't web scrapers just pretend to be Lynx browsers?

Some do.  That's why so many web servers refuse connections from Lynx.
--scott
-- 
"C'est un Nagra. C'est suisse, et tres, tres precis."

[toc] | [prev] | [next] | [standalone]

#26422

On Wed, 29 Jan 2025, Scott Dorsey wrote:

> In article <861pwl5qu8.fsf@example.com>, Richmond  <dnomhcir@gmx.com> wrote:
>>
>> Why can't web scrapers just pretend to be Lynx browsers?
>
> Some do.  That's why so many web servers refuse connections from Lynx.
> --scott

That's racism and illegal! I use elinks and have not had any problems. It 
must be the Trump of text based browsers!

[toc] | [prev] | [next] | [standalone]

#26423

>>>>> On 2025-01-29, Scott Dorsey wrote:
>>>>> Richmond <dnomhcir@gmx.com> wrote:

 >> Why can't web scrapers just pretend to be Lynx browsers?

 > Some do.  That's why so many web servers refuse connections from Lynx.

	IME it's more common for HTTP servers to react to "libwww" in
	Lynx' User-Agent: rather than "Lynx": removing the former (while
	keeping "Lynx") have often enough resolved the issue for me.

	(These days, I mostly just switch to reading the site via
	http://web.archive.org/ right away, though.)

	Might be because "libwww" is both the name of the library Lynx
	is based on, /and/ the name of an unrelated (AIUI) Perl library
	that, I gather, used to be popular among web robot writers.
	(See, e. g., http://packages.debian.org/sid/libwww-perl .)

	A cursory look over my access.log files seems to hint that Go
	is way more popular a choice for the task these days, though
	my overall impression is that robot authors just use any of
	the popular user agent strings for their software instead of
	anything that might identify their actual codebase.

	Which means that making *any* big decisions based on User-Agent:
	statistics (like, "Look, we're getting lots of hits from
	Arachne users recently; let's optimize our site for their
	best experience at once!") is ill-advised at best: you might
	end up being trolled by a particularly creative botnet operator.

	Personally, as a web author, I try to a. stick to the standards;
	b. have an actual reason for using one feature or another
	(rather than going for "for consistency" or "just because" or
	"this new shiny framework needs it") [*]; and c. mind my audience.

	Sure, I use Lynx a lot for testing, so the webpages I author
	tend to end up being compatible with Lynx, and might be less
	compatible with other UAs.  However, the idea that I should
	adapt my practices to the idiosynchrasies of any particular
	UA, regardless of its market share, rubs me the wrong way.
	The "making sure the site works with IE" sort of wrong.

	Conversely, as a reader of that same web, I expect to get a
	standards-compliant document from the site.  I deem it my own
	responsibility to make use of it.  For instance, I certainly
	won't hold it against the site operator if /my/ software chokes
	on something that /is/ standard.

	What really irks me, though, is when in place of a document,
	I get an application.  (Doesn't even matter if it's .js, .exe,
	or .tex.)

	Not that I don't get disappointed on occasion when a website
	"improves" its typography, or switches to a more "mobile-friendly"
	look and feel.  But that's one of the major reasons for me to
	stick with Lynx in the first place: go and try to tweak the CSS
	to make your website look more "modern" when viewed with Lynx!

   [*]	As a rule, my HTML is expected to comply with the requirements
	of the Live Standard, for both text/html and application/xml+xhtml
	Content-Type:s at the same time (the idea is that if .xhtml does
	not work for someone, the file can be downloaded, renamed to
	.html, and viewed that way.)  My CSS should be /mostly/ 2.1
	with some CSS3 Selectors (though I haven't quite checked it.)
	When JavaScript is used (i. e., when I publish an application,
	not just a document), it's ought to conform to ECMA-262 6 (2015),
	though the set of browser APIs used might vary depending on what
	the application aims to do.

[toc] | [prev] | [next] | [standalone]

#26418

Richmond <dnomhcir@gmx.com> wrote:
> Rich <rich@example.invalid> writes:
> 
>> candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> wrote:
>>> Salvador Mirzo <smirzo@example.com> wrote at 13:33 this Tuesday
>>>(GMT): > Sylvia Else <sylvia@email.invalid> writes:
>>>>
>>>>> On 21-Jan-25 5:10 am, Computer Nerd Kev wrote:
>>>>>> Google begins requiring JavaScript for Google Search by Thom
>>>>>>   Holwerda 2025-01-18 -
>>>>>>   https://www.osnews.com/story/141570/google-begins-requiring-javascript-for-google-search/
>>>>>>   " Google says it has begun requiring users to turn on
>>>>>>   JavaScript, the widely used programming language to make web
>>>>>>   pages interactive, in order to use Google Search.  In an email
>>>>>>   to TechCrunch, a company spokesperson claimed that the change is
>>>>>>   intended to "better protect" Google Search against malicious
>>>>>>   activity, such as bots and spam, and to improve the overall
>>>>>>   Google Search experience for users.
>>>>
>>>> [...]
>>>>
>>>>> How is this going to '"better protect" Google Search against
>>>>> malicious activity, such as bots and spam'?
>>>>
>>>> I believe the idea is that if the robot doesn't speak Javascript,
>>>> it's an easy denial by the web server.  And making bots speak
>>>> Javascript is one step up.  And with Javascript they can likely
>>>> monitor things like mouse movement to detect whether the user is a
>>>> human or a robot.
>>>>
>>>> I'm not approving the idea; just sharing what I think they might
>>>> have in mind when they say Javascript will help them fend off
>>>> robots.
>>> 
>>> It would also make it harder to scrape, since I /think/ web scrapers
>>> don't run JS by default.
>>
>> Which just means this will push web scrapers to start running JS.
>>
>> They don't run JS (yet) because they have not needed to run JS to do
>> their scraping.  But if JS is required, and they want to scrape bad
>> enough, they will put in support for running JS.
> 
> Why can't web scrapers just pretend to be Lynx browsers?

They could, that is until google simply starts expecting JS to be 
executed regardless of the value of the user agent header.

[toc] | [prev] | [next] | [standalone]

#26419

Rich <rich@example.invalid> wrote:

> They could, that is until google simply starts expecting JS to be
> executed regardless of the value of the user agent header.

Then scrapers will add JS to their agents and the users of older
browsers are the only ones reliably locked out.

-- 
"The government you elect is the government you deserve"
                                                      - Thomas Jefferson

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.misc

csiph-web