Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #229307

Web-bot tarpit aka spider trap (was: swamp rat bots Q)

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

hdv@gmail (12020-12-04):
> Let me offer you an alternative option. (Most) bots work by analysing the
> referrals on each page in your website. Right? So, why not add a link to a
> page that normal users will never visit (e.g. because they do not see the
> link and thus will never click on it), but will show up in a bot's analysis?
> That way you can monitor your logs for entries containing that page. Every
> entity requesting that specific URL is blocked.

This made me think of something.

A long time ago, a friend of mine implemented, to trap the badly-behaved
robots, something called the Book of Infinity: a set of deterministic
pseud-random pages linking to sub-pages ad infinitum, with ever growing
URLs.

As it happened, it was not actually a good idea, and released a lot of
CO₂, and the very badly behaved robots had to be blacklisted from
explring it. (At some point, we had the same problem when Googlebot
tried to brute-force our online make-your-own-adventure book, but
Googlebot heeds robots.txt.)

But it could be coupled with techniques inspired by spam tarpits: have
the server reply at a crawl to force the bots to waste resources, while
keeping the resource consumption on the server strictly bounded.

Oh, I just noticed I was not the first one to think of it: Wikipedia
tells me it's called a spider trap.

https://en.wikipedia.org/wiki/Spider_trap

Regards,

-- 
  Nicolas George

Back to linux.debian.user | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 14:00 +0100
  Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-03 14:10 +0100
    Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 15:10 +0100
      Re: swamp rat bots Q Greg Wooledge <wooledg@eeg.ccf.org> - 2020-12-03 15:20 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 20:00 +0100
          Re: swamp rat bots Q Håkon Alstadheim <hakon@alstadheim.priv.no> - 2020-12-03 22:00 +0100
          Re: swamp rat bots Q Keith Christian <keith1christian@gmail.com> - 2020-12-03 22:30 +0100
      Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-03 16:40 +0100
  Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 03:20 +0100
    Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 06:10 +0100
      Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 07:10 +0100
        Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-04 07:40 +0100
          Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 08:00 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 07:50 +0100
          Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-04 08:20 +0100
          Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 08:50 +0100
      Re: swamp rat bots Q "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2020-12-04 11:50 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 14:50 +0100
      Re: swamp rat bots Q Andrei POPESCU <andreimpopescu@gmail.com> - 2020-12-04 12:00 +0100
  Re: swamp rat bots Q "hdv@gmail" <hdv.jadev@gmail.com> - 2020-12-04 10:00 +0100
    Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 14:50 +0100
      Re: swamp rat bots Q Reco <recoverym4n@enotuniq.net> - 2020-12-04 18:40 +0100
        Re: swamp rat bots Q grumpy@mailfence.com - 2020-12-04 19:10 +0100
          Re: swamp rat bots Q Reco <recoverym4n@enotuniq.net> - 2020-12-04 19:20 +0100
            Re: swamp rat bots Q Carl Fink <carlf@panix.com> - 2020-12-04 19:30 +0100
            Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 21:00 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 21:00 +0100
          Re: swamp rat bots Q Tixy <tixy@yxit.co.uk> - 2020-12-04 22:20 +0100
            Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 23:10 +0100
              Re: swamp rat bots Q Tixy <tixy@yxit.co.uk> - 2020-12-04 23:40 +0100
                Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-05 01:00 +0100
                Re: swamp rat bots Q Andrei POPESCU <andreimpopescu@gmail.com> - 2020-12-05 11:00 +0100
                Re: swamp rat bots Q elvis <elvis@dogonfire.com> - 2020-12-05 01:50 +0100
                Re: swamp rat bots Q grumpy@mailfence.com - 2020-12-05 02:00 +0100
    Web-bot tarpit aka spider trap (was: swamp rat bots Q) Nicolas George <george@nsup.org> - 2020-12-04 15:10 +0100
      Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Gene Heskett <gheskett@shentel.net> - 2020-12-04 17:10 +0100
        Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) <tomas@tuxteam.de> - 2020-12-04 22:10 +0100
          Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) "Martin McCormick" <martin.m@suddenlink.net> - 2020-12-06 20:20 +0100
            Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) David Christensen <dpchrist@holgerdanske.com> - 2020-12-06 21:30 +0100
            Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Charles Curley <charlescurley@charlescurley.com> - 2020-12-06 23:10 +0100
              Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Gene Heskett <gheskett@shentel.net> - 2020-12-07 02:30 +0100
                Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Doug McGarrett <dmcgarrett@optonline.net> - 2020-12-07 04:00 +0100

csiph-web