Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #261759 > unrolled thread

[a bit OT] Automate a (G o o g l e) search from a list of strings

Back to article view | Back to linux.debian.user

Contents

  [a bit OT] Automate a (G o o g l e) search from a list of strings steve <dlist@bluewin.ch> - 2023-09-19 16:50 +0200
    Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-19 16:50 +0200
    Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Nicolas George <george@nsup.org> - 2023-09-19 17:00 +0200
      Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Jason <jason@7748229.xyz> - 2023-09-19 19:30 +0200
      Re: [a bit OT] Automate a (G o o g l e) search from a list of strings steve <dlist@bluewin.ch> - 2023-09-20 08:20 +0200
        Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Andy Smith <andy@strugglers.net> - 2023-09-20 10:50 +0200
          Re: [a bit OT] Automate a (G o o g l e) search from a list of strings steve <dlist@bluewin.ch> - 2023-09-20 11:40 +0200
          Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Tom Browder <tom.browder@gmail.com> - 2023-09-20 17:50 +0200
            Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Nicolas George <george@nsup.org> - 2023-09-20 20:40 +0200
              Re: [a bit OT] Automate a (G o o g l e) search from a list of strings <tomas@tuxteam.de> - 2023-09-20 21:20 +0200
              Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Tom Browder <tom.browder@gmail.com> - 2023-09-20 23:00 +0200
                Re: [a bit OT] Automate a (G o o g l e) search from a list of strings The Wanderer <wanderer@fastmail.fm> - 2023-09-21 00:00 +0200
                  Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Tamas Sohamar <sohamartamas@gmail.com> - 2023-09-21 07:10 +0200
    Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Karl Vogel <vogelke@pobox.com> - 2023-09-20 04:10 +0200
      Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Greg Wooledge <greg@wooledge.org> - 2023-09-20 04:40 +0200

#261759 — [a bit OT] Automate a (G o o g l e) search from a list of strings

Hello,

I'm sorry if this question is a bit OT but since the answer will be
implemented from a Debian machine, it's not completely OT :)

I have a list of 200 keywords and would like for every one to launch a
search on a specific website and put the result(s) in a file, something
like:

search keyword website: example.com >> file.csv

I guess I could code a Python script to do that but if something already
exists I'd rather use it.

Any ideas or links ?

Thanks !

s.

[toc] | [next] | [standalone]

#261760

On 19 Sep 2023 16:41 +0200, from dlist@bluewin.ch (steve):
> I have a list of 200 keywords and would like for every one to launch a
> search on a specific website and put the result(s) in a file, something
> like:
> 
> search keyword website: example.com >> file.csv
> 
> I guess I could code a Python script to do that but if something already
> exists I'd rather use it.

Look at pup.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[toc] | [prev] | [next] | [standalone]

#261761

steve (12023-09-19):
> I have a list of 200 keywords and would like for every one to launch a
> search on a specific website and put the result(s) in a file, something
> like:
> 
> search keyword website: example.com >> file.csv
> 
> I guess I could code a Python script to do that but if something already
> exists I'd rather use it.
> 
> Any ideas or links ?

Hi.

If you have access to Google's APIs, then the answer is probably in
their documentation.

If not, then what you intend is completely forbidden by Google's terms
and service. And they have detection: please only do this on a computer
and network access when you will be the only one inconvenienced when
they block your access. It happened on a computer I co-administrate.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#261764

[Multipart message — attachments visible in raw view] — view raw

On 19.09.23 16:52, Nicolas George wrote:
> If not, then what you intend is completely forbidden by Google's terms
> and service. And they have detection: please only do this on a computer
> and network access when you will be the only one inconvenienced when
> they block your access. It happened on a computer I co-administrate.

I wouldn't query Google directly. They are evil, as already mentioned.

Use an alternative front-end, for example https://searx.be
Source and list of public instances https://github.com/searxng/searxng

cheers
Jason

[toc] | [prev] | [next] | [standalone]

#261769

Dear all,

Thank you for your answers, unfortunately they don't help me much
(provided code is too complicated for me).


Le 19-09-2023, à 16:52:24 +0200, Nicolas George a écrit :

>steve (12023-09-19):
>> I have a list of 200 keywords and would like for every one to launch a
>> search on a specific website and put the result(s) in a file, something
>> like:
>>
>> search keyword website: example.com >> file.csv
>>
>> I guess I could code a Python script to do that but if something already
>> exists I'd rather use it.
>>
>> Any ideas or links ?
>
>Hi.
>
>If you have access to Google's APIs, then the answer is probably in
>their documentation.

I don't.

>If not, then what you intend is completely forbidden by Google's terms
>and service. And they have detection: please only do this on a computer
>and network access when you will be the only one inconvenienced when
>they block your access. It happened on a computer I co-administrate.

Why would it be forbidden, I just want to automate what I could do
manually? Nothing evil in my approach.

Have a nice day.

s.

[toc] | [prev] | [next] | [standalone]

#261770

Hello,

On Wed, Sep 20, 2023 at 08:13:43AM +0200, steve wrote:
> Le 19-09-2023, à 16:52:24 +0200, Nicolas George a écrit :
> > what you intend is completely forbidden by Google's terms and
> > service. And they have detection: please only do this on a
> > computer and network access when you will be the only one
> > inconvenienced when they block your access. It happened on a
> > computer I co-administrate.
> 
> Why would it be forbidden, I just want to automate what I could do
> manually? Nothing evil in my approach.

Google terms of use forbid mechanization, automation and scraping of
their service. And as Nicolas says, they do check, and they do
punish offenders. At best you'd be hoping that the volume of your
queries falls below their radar. But I wouldn't risk it.

A typical punishment is to require a CAPTCHA be solved for any query
from your netblock to any Google property, which is immensely
annoying.

I have had to terminate my own customers on multiple occasions over
their inability to take this exact issue seriously, because it
really upsets OTHER customers when they suddenly need to solve a
CAPTCHA to do a search query or look at their email.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#261771

Le 20-09-2023, à 08:46:06 +0000, Andy Smith a écrit :

>Hello,
>
>On Wed, Sep 20, 2023 at 08:13:43AM +0200, steve wrote:
>> Le 19-09-2023, à 16:52:24 +0200, Nicolas George a écrit :
>> > what you intend is completely forbidden by Google's terms and
>> > service. And they have detection: please only do this on a
>> > computer and network access when you will be the only one
>> > inconvenienced when they block your access. It happened on a
>> > computer I co-administrate.
>>
>> Why would it be forbidden, I just want to automate what I could do
>> manually? Nothing evil in my approach.
>
>Google terms of use forbid mechanization, automation and scraping of
>their service. And as Nicolas says, they do check, and they do
>punish offenders. At best you'd be hoping that the volume of your
>queries falls below their radar. But I wouldn't risk it.
>
>A typical punishment is to require a CAPTCHA be solved for any query
>from your netblock to any Google property, which is immensely
>annoying.
>
>I have had to terminate my own customers on multiple occasions over
>their inability to take this exact issue seriously, because it
>really upsets OTHER customers when they suddenly need to solve a
>CAPTCHA to do a search query or look at their email.

Thank you very much for this very clear explanation, I didn't know that
(never read those terms of use to be honest).

I ended up doing my searches by hand, it took some time and that's all
right.

Best,

s.

[toc] | [prev] | [next] | [standalone]

#261777

[Multipart message — attachments visible in raw view] — view raw

On Wed, Sep 20, 2023 at 09:35 Andy Smith <andy@strugglers.net> wrote:

> Hello,
>
> On Wed, Sep 20, 2023 at 08:13:43AM +0200, steve wrote:
> > Le 19-09-2023, à 16:52:24 +0200, Nicolas George a écrit :
> > > what you intend is completely forbidden by Google's terms and
> > > service. And they have detection: please only do this on a
> > > computer and network access when you will be the only one
> > > inconvenienced when they block your access. It happened on a
> > > computer I co-administrate.


What if you used an equilavent script but increased and randomized time
between each search string?  Or do you think just the single search is
enough to trigger them?

-Tom

[toc] | [prev] | [next] | [standalone]

#261789

Tom Browder (12023-09-20):
> What if you used an equilavent script but increased and randomized time
> between each search string?  Or do you think just the single search is
> enough to trigger them?

We can try to exercise some common sense, in particular by comparing to
similar situations. For example, if you take something that does not
belong to you, but do it at night, when everybody is sleeping and being
very careful you do not make a step squeak or break the laser beams, is
it still stealing? I think most people would consider the answer to the
question whether what you describes breaks Google's terms of use is the
same.

Now, if your interrogation is whether we think you would get away with
it. Well, as it is roughly equivalent to asking if I think you are
smarter than the people at Google, I will respectfully decline to
answer.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#261791

[Multipart message — attachments visible in raw view] — view raw

On Wed, Sep 20, 2023 at 08:35:54PM +0200, Nicolas George wrote:
> Tom Browder (12023-09-20):
> > What if you used an equilavent script but increased and randomized time
> > between each search string?  Or do you think just the single search is
> > enough to trigger them?
> 
> We can try to exercise some common sense, in particular by comparing to
> similar situations. For example, if you take something that does not
> belong to you, but do it at night, when everybody is sleeping and being
> very careful you do not make a step squeak or break the laser beams, is
> it still stealing? I think most people would consider the answer to the
> question whether what you describes breaks Google's terms of use is the
> same.
> 
> Now, if your interrogation is whether we think you would get away with
> it. Well, as it is roughly equivalent to asking if I think you are
> smarter than the people at Google, I will respectfully decline to
> answer.

Or possibly Google doesn't care /that/ much. Or even possibly they
welcome the degree of lock-in they achieve in "markets" which usually
are beyond their reach (there was a time when Microsoft was definitely
doing this: their license numbers were so easily guessable that it
almost surely wasn't a honest mistake).

Reality is often even messier :)

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#261792

[Multipart message — attachments visible in raw view] — view raw

 On Wed, Sep 20, 2023 at 13:36 Nicolas George <george@nsup.org> wrote:

> Tom Browder (12023-09-20):
> > What if you used an equilavent script but increased and randomized time

...

We can try to exercise some common sense, in particular by comparing to
> similar situations. For example, if you take something that does not
> belong to you, but do it at night, when everybody is sleeping and being
> very careful you do not make a step squeak or break the laser beams, is
> it still stealing?


I apologize. I was not referring to stealing, and I haven't read the
details in the terms of use. What I should have  asked was: "is a single
query in the script okay?" If so, how much time would have to pass before
the next query in order to adhere to the terms of service?

In the distant past I have used Google's APIs to search mail,  and I
believe there was some kind of rate or time limit for their use.

-Tom

[toc] | [prev] | [next] | [standalone]

#261794

[Multipart message — attachments visible in raw view] — view raw

On 2023-09-20 at 16:50, Tom Browder wrote:

> On Wed, Sep 20, 2023 at 13:36 Nicolas George <george@nsup.org>
> wrote:
> 
>> Tom Browder (12023-09-20):
>> 
>>> What if you used an equilavent script but increased and
>>> randomized time
> 
> ...
> 
>> We can try to exercise some common sense, in particular by
>> comparing to similar situations. For example, if you take something
>> that does not belong to you, but do it at night, when everybody is
>> sleeping and being very careful you do not make a step squeak or
>> break the laser beams, is it still stealing?
> 
> I apologize. I was not referring to stealing, and I haven't read the 
> details in the terms of use. What I should have  asked was: "is a
> single query in the script okay?" If so, how much time would have to
> pass before the next query in order to adhere to the terms of
> service?

You'd have to refer to the TOS to be certain, but based on the way
they've been described here, it isn't a question of amount of time. It's
the fact that you're applying scripts and automation at all, vs. having
each search be individually triggered by (and, I suspect, the results
sent/shown to) a real-person user.

Yes, this is horrific in light of the fact that automation is what
computers are *for* - but it also makes sense from Google's likely
perspective, and it's entirely plausible that they might care enough to
try to detect even minimal amounts of automation.

In principle, it probably *would* be possible to script doing such a
thing in such a way that it would not be sufficiently distinguishable
from human access patterns for them to be able to tell. (Varying time
between accesses almost certainly would not do it; among possibly other
things, you'd need to specify the accessing program, etc., in a way that
is doable but not necessarily obvious.)

That said, you'd be taking a gamble that your ability to obfuscate such
things is better than Google's ability to detect it. Who would you
prefer to bet on in a competition like that?

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#261798

[Multipart message — attachments visible in raw view] — view raw

Using another search engine is not an option?

On Thu, 21 Sept 2023, 05:55 The Wanderer, <wanderer@fastmail.fm> wrote:

> On 2023-09-20 at 16:50, Tom Browder wrote:
>
> > On Wed, Sep 20, 2023 at 13:36 Nicolas George <george@nsup.org>
> > wrote:
> >
> >> Tom Browder (12023-09-20):
> >>
> >>> What if you used an equilavent script but increased and
> >>> randomized time
> >
> > ...
> >
> >> We can try to exercise some common sense, in particular by
> >> comparing to similar situations. For example, if you take something
> >> that does not belong to you, but do it at night, when everybody is
> >> sleeping and being very careful you do not make a step squeak or
> >> break the laser beams, is it still stealing?
> >
> > I apologize. I was not referring to stealing, and I haven't read the
> > details in the terms of use. What I should have  asked was: "is a
> > single query in the script okay?" If so, how much time would have to
> > pass before the next query in order to adhere to the terms of
> > service?
>
> You'd have to refer to the TOS to be certain, but based on the way
> they've been described here, it isn't a question of amount of time. It's
> the fact that you're applying scripts and automation at all, vs. having
> each search be individually triggered by (and, I suspect, the results
> sent/shown to) a real-person user.
>
> Yes, this is horrific in light of the fact that automation is what
> computers are *for* - but it also makes sense from Google's likely
> perspective, and it's entirely plausible that they might care enough to
> try to detect even minimal amounts of automation.
>
> In principle, it probably *would* be possible to script doing such a
> thing in such a way that it would not be sufficiently distinguishable
> from human access patterns for them to be able to tell. (Varying time
> between accesses almost certainly would not do it; among possibly other
> things, you'd need to specify the accessing program, etc., in a way that
> is doable but not necessarily obvious.)
>
> That said, you'd be taking a gamble that your ability to obfuscate such
> things is better than Google's ability to detect it. Who would you
> prefer to bet on in a competition like that?
>
> --
>    The Wanderer
>
> The reasonable man adapts himself to the world; the unreasonable one
> persists in trying to adapt the world to himself. Therefore all
> progress depends on the unreasonable man.         -- George Bernard Shaw
>
>

[toc] | [prev] | [next] | [standalone]

#261765

On Tue, Sep 19, 2023 at 10:42:14AM -0400, steve wrote:
> I have a list of 200 keywords and would like for every one to launch a
> search on a specific website and put the result(s) in a file [...]

Take a list of words and turn it into a single Google query matching any
of them:

    me% cat keywords
    corsairs
    buccaneers
    privateers

Desired query:

    https://www.google.com/search?q=corsairs+OR+buccaneers+OR+privateers

Script:

    me% cat search
    #!/bin/sh
    export PATH=/usr/local/bin:/bin:/usr/bin
    set -o nounset
    umask 022

    query=$(tr "[:space:]" "+" < keywords |
            sed -e 's/ /+OR+/g' -e 's/+$//' -e 's/+/+OR+/g')

    curl -s -L -o pirate.htm "https://www.google.com/search?q=${query}"
    ls -l pirate.htm
    exit 0

Results:

    me% ./search
    -rw-r--r-- 1 vogelke 220487 19-Sep-2023 21:33:21 pirate.htm

If you want results in text form, lots of programs can do that:

    me% lynx -cookies -accept_all_cookies -dump -width 80 pirate.htm
    Tampa Bay Buccaneers: Buccaneers Home
    www.buccaneers.com
    The official source of the latest Bucs headlines, news, videos, photos,
    tickets, rosters, stats, schedule and gameday information.
  
    People also ask
    What is the difference between a pirate and a corsair?
    What kind of pirate is a corsair?
    What were Spanish pirates called?
    What is the history of corsairs?

    Pirates, Privateers, Corsairs, Buccaneers: What's the Difference?
    www.britannica.com > Demystified > Geography & Travel
    Corsairs were essentially privateers, although the term corsair carried
    an added religious connotation because the conflict was between Muslim
    and Christian...

HTH.

-- 
Karl Vogel                      I don't speak for the USAF or my company

Leonardo diCaprio in Titanic: Despite acting like a plank of wood,
he still can't float.  --"Top Ten Most Welcome Movie Deaths", Stylus Magazine

[toc] | [prev] | [next] | [standalone]

#261766

On Tue, Sep 19, 2023 at 09:53:25PM -0400, Karl Vogel wrote:
> On Tue, Sep 19, 2023 at 10:42:14AM -0400, steve wrote:
> > I have a list of 200 keywords and would like for every one to launch a
> > search on a specific website and put the result(s) in a file [...]
> 
> Take a list of words and turn it into a single Google query matching any
> of them:
> 
>     me% cat keywords
>     corsairs
>     buccaneers
>     privateers
> 
> Desired query:
> 
>     https://www.google.com/search?q=corsairs+OR+buccaneers+OR+privateers

It sounds more like steve wanted 200 separate searches, one for each
keyword.  For that, you'd use a very simple loop.

> Script:
> 
>     me% cat search
>     #!/bin/sh
>     export PATH=/usr/local/bin:/bin:/usr/bin
>     set -o nounset
>     umask 022
> 
>     query=$(tr "[:space:]" "+" < keywords |
>             sed -e 's/ /+OR+/g' -e 's/+$//' -e 's/+/+OR+/g')
> 
>     curl -s -L -o pirate.htm "https://www.google.com/search?q=${query}"
>     ls -l pirate.htm
>     exit 0

Joining a list (array) of strings into a big string is so horribly ugly
in /bin/sh.  Switching to bash buys you so many better options.

#!/bin/bash
mapfile -t words < keywords
printf -v query '%s+OR+' "${words[@]}"
query=${query%+OR+}
curl -s -L -o output.html "https://www.google.com/search?q=$query"

Of course, if you were writing this for publication, you'd want to
URL-encode the keywords themselves, because they might contain spaces,
plus signs, ampersands, etc.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web