Re: Crawl the Web Fast From the Terminal

From	NerdRat Hispagatos <NerdRat@chocolate.cookies>
Newsgroups	alt.comp.lang.rust
Subject	Re: Crawl the Web Fast From the Terminal
Date	2025-03-15 17:27 +0000
Organization	Hispagatos
Message-ID	<vr4d9f$2mlqb$1@matrix.hispagatos.org> (permalink)
References	<vqdaeb$3n0k7$1@matrix.hispagatos.org> <vqf00j$36dc$1@matrix.hispagatos.org>

Show all headers | View raw

On 2025-03-07, rek2 hispagatos <rek2@hispagatos.org.invalid> wrote:
> On 2025-03-06, NerdRat Hispagatos <NerdRat@cookies.chocolate> wrote:
>> I always wanted to crawl the web from the terminal in the most
>> transparent way possible. Html makes this almost unreadable, but with a
>> scraper it gets easier.
>>
>> Traditionaly i did this with curl and grep with extended regex under a
>> bash script, but looking at the rust daily mastodon accoutn i found
>> "is-fast"; which "is a TUI tool designed for quick and efficient
>> internet searches directly from the terminal, ideal for environments
>> where you don't have easy access to a browser"
>>
>> or you don't want to use the browser.
>>
>> See it yourself:
>>
>> https://github.com/Magic-JD/is-fast
>>
>> There are more demos here:
>>
>> https://github.com/Magic-JD/is-fast/blob/main/demos/DEMOS.md
>>
>> Just clone the repo and `cargo build -r` or 
>>
>> ```
>> cargo install --git https://github.com/Magic-JD/is-fast.git
>> ```
>
>   This is super interesting, what search enginer does it use as backend?
>   reason i ask is that I am not home today, if not I will just had look
>   it up myself.

Sorry for not answering soon, a lot of sudy going on lately.

I know you may have already found out by yourself but default search
engine is duckduckgo, but supports google and kagi (don't know much
about it). But you can add your custom search engine by forking the
repo and hack on it,

https://github.com/Magic-JD/is-fast?tab=readme-ov-file#-search-configuration

I wish this was made a hybrid package so we could use it as a library
and embed it into other programs.

-- 
NerdRat

Back to alt.comp.lang.rust | Previous | Next — Previous in thread | Find similar

Thread

Crawl the Web Fast From the Terminal NerdRat Hispagatos <NerdRat@cookies.chocolate> - 2025-03-06 23:17 +0000
  Re: Crawl the Web Fast From the Terminal rek2 hispagatos <rek2@hispagatos.org.invalid> - 2025-03-07 14:31 +0000
    Re: Crawl the Web Fast From the Terminal NerdRat Hispagatos <NerdRat@chocolate.cookies> - 2025-03-15 17:27 +0000

csiph-web