Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #261766

Re: [a bit OT] Automate a (G o o g l e) search from a list of strings

Show all headers | View raw

On Tue, Sep 19, 2023 at 09:53:25PM -0400, Karl Vogel wrote:
> On Tue, Sep 19, 2023 at 10:42:14AM -0400, steve wrote:
> > I have a list of 200 keywords and would like for every one to launch a
> > search on a specific website and put the result(s) in a file [...]
> 
> Take a list of words and turn it into a single Google query matching any
> of them:
> 
>     me% cat keywords
>     corsairs
>     buccaneers
>     privateers
> 
> Desired query:
> 
>     https://www.google.com/search?q=corsairs+OR+buccaneers+OR+privateers

It sounds more like steve wanted 200 separate searches, one for each
keyword.  For that, you'd use a very simple loop.

> Script:
> 
>     me% cat search
>     #!/bin/sh
>     export PATH=/usr/local/bin:/bin:/usr/bin
>     set -o nounset
>     umask 022
> 
>     query=$(tr "[:space:]" "+" < keywords |
>             sed -e 's/ /+OR+/g' -e 's/+$//' -e 's/+/+OR+/g')
> 
>     curl -s -L -o pirate.htm "https://www.google.com/search?q=${query}"
>     ls -l pirate.htm
>     exit 0

Joining a list (array) of strings into a big string is so horribly ugly
in /bin/sh.  Switching to bash buys you so many better options.

#!/bin/bash
mapfile -t words < keywords
printf -v query '%s+OR+' "${words[@]}"
query=${query%+OR+}
curl -s -L -o output.html "https://www.google.com/search?q=$query"

Of course, if you were writing this for publication, you'd want to
URL-encode the keywords themselves, because they might contain spaces,
plus signs, ampersands, etc.

Back to linux.debian.user | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

[a bit OT] Automate a (G o o g l e) search from a list of strings steve <dlist@bluewin.ch> - 2023-09-19 16:50 +0200
  Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Michael Kjörling <2695bd53d63c@ewoof.net> - 2023-09-19 16:50 +0200
  Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Nicolas George <george@nsup.org> - 2023-09-19 17:00 +0200
    Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Jason <jason@7748229.xyz> - 2023-09-19 19:30 +0200
    Re: [a bit OT] Automate a (G o o g l e) search from a list of strings steve <dlist@bluewin.ch> - 2023-09-20 08:20 +0200
      Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Andy Smith <andy@strugglers.net> - 2023-09-20 10:50 +0200
        Re: [a bit OT] Automate a (G o o g l e) search from a list of strings steve <dlist@bluewin.ch> - 2023-09-20 11:40 +0200
        Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Tom Browder <tom.browder@gmail.com> - 2023-09-20 17:50 +0200
          Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Nicolas George <george@nsup.org> - 2023-09-20 20:40 +0200
            Re: [a bit OT] Automate a (G o o g l e) search from a list of strings <tomas@tuxteam.de> - 2023-09-20 21:20 +0200
            Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Tom Browder <tom.browder@gmail.com> - 2023-09-20 23:00 +0200
              Re: [a bit OT] Automate a (G o o g l e) search from a list of strings The Wanderer <wanderer@fastmail.fm> - 2023-09-21 00:00 +0200
                Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Tamas Sohamar <sohamartamas@gmail.com> - 2023-09-21 07:10 +0200
  Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Karl Vogel <vogelke@pobox.com> - 2023-09-20 04:10 +0200
    Re: [a bit OT] Automate a (G o o g l e) search from a list of strings Greg Wooledge <greg@wooledge.org> - 2023-09-20 04:40 +0200

csiph-web