Re: swamp rat bots Q

From	Gene Heskett <gheskett@shentel.net>
Newsgroups	linux.debian.user
Subject	Re: swamp rat bots Q
Date	2020-12-04 23:10 +0100
Message-ID	<Birr4-2Qp-23@gated-at.bofh.it> (permalink)
References	<BhWnf-85-7@gated-at.bofh.it> <Bippf-13b-5@gated-at.bofh.it> <BiqEG-24y-17@gated-at.bofh.it>
Organization	none,nada,zip

Show all headers | View raw

On Friday 04 December 2020 16:14:29 Tixy wrote:

> On Fri, 2020-12-04 at 14:51 -0500, Gene Heskett wrote:
> > On Friday 04 December 2020 12:39:24 Reco wrote:
> > >       Hi.
> > >
> > > On Fri, Dec 04, 2020 at 08:39:42AM -0500, Gene Heskett wrote:
> > > > But I asked specifically how to enable it for one bot, and I've
> > > > asked that question several times, getting smoke and mirror
> > > > answers you all assume are helpfull, but which are useless to a
> > > > new user installing the now 7 years old and long out of date
> > > > package that in effect has no "how it works" docs. I asked 3
> > > > questions in a previous day or so timeline, and no one has
> > > > actually attempted to actually answer even one of them. Here is
> > > > one line from that log: and that I just blocked:
> > > >
> > > > coyote.coyote.den:80 192.99.6.226 - -
> > > > [04/Dec/2020:07:18:20 -0500] "GET
> > > > /gene/toolshed/c3/build/win32/prep/?C=S;O=D HTTP/1.1" 200 673
> > > > "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8;
> > > > http://mj12bot.com/)"
> > >
> > > Taken directly from the link.
> > >
> > > Bot Type         Good crawler (always identifies itself)
> > > IP Range         Distributed, Worldwide
> > > Obeys Robots.txt *Yes*
> >
> > Sorry, they do not, they've read it and ignored it 428 times in the
> > life of that log which I zeroed out around 1 July of this year.
>
> Why would they read it if they we're going to just ignore it, perhaps
> your robots.txt is broken? Hint, it is, in 2 or 3 different ways I can
> see (if it's http://geneslinuxbox.net:6309/robots.txt we're talking
> about). That file doesn't have any syntactically correct entry in
> there for blocking that bot.

And what might that be like, I'll fix it right now

> I don't know why you seem set on blaming malice on part of a bot whose
> front web page has sections like:

The evidence I have collected so far indicates they don't care who they 
ddos with their repeated suckage.  So I never considered allowing their 
site anything like direct access by going to it.  Their actions speak 
MUCH louder than the words below.

>    How can I block MJ12bot?
>    How can I slow down MJ12bot?
>    What commands in robots.txt does MJ12bot support?
>    Why did my robots.txt block not work on MJ12bot?
>
> The URL for that page is in the user-agent string from the log snippet
> you posted above.


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Thread

swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 14:00 +0100
  Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-03 14:10 +0100
    Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 15:10 +0100
      Re: swamp rat bots Q Greg Wooledge <wooledg@eeg.ccf.org> - 2020-12-03 15:20 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 20:00 +0100
          Re: swamp rat bots Q Håkon Alstadheim <hakon@alstadheim.priv.no> - 2020-12-03 22:00 +0100
          Re: swamp rat bots Q Keith Christian <keith1christian@gmail.com> - 2020-12-03 22:30 +0100
      Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-03 16:40 +0100
  Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 03:20 +0100
    Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 06:10 +0100
      Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 07:10 +0100
        Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-04 07:40 +0100
          Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 08:00 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 07:50 +0100
          Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-04 08:20 +0100
          Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 08:50 +0100
      Re: swamp rat bots Q "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2020-12-04 11:50 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 14:50 +0100
      Re: swamp rat bots Q Andrei POPESCU <andreimpopescu@gmail.com> - 2020-12-04 12:00 +0100
  Re: swamp rat bots Q "hdv@gmail" <hdv.jadev@gmail.com> - 2020-12-04 10:00 +0100
    Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 14:50 +0100
      Re: swamp rat bots Q Reco <recoverym4n@enotuniq.net> - 2020-12-04 18:40 +0100
        Re: swamp rat bots Q grumpy@mailfence.com - 2020-12-04 19:10 +0100
          Re: swamp rat bots Q Reco <recoverym4n@enotuniq.net> - 2020-12-04 19:20 +0100
            Re: swamp rat bots Q Carl Fink <carlf@panix.com> - 2020-12-04 19:30 +0100
            Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 21:00 +0100
        Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 21:00 +0100
          Re: swamp rat bots Q Tixy <tixy@yxit.co.uk> - 2020-12-04 22:20 +0100
            Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 23:10 +0100
              Re: swamp rat bots Q Tixy <tixy@yxit.co.uk> - 2020-12-04 23:40 +0100
                Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-05 01:00 +0100
                Re: swamp rat bots Q Andrei POPESCU <andreimpopescu@gmail.com> - 2020-12-05 11:00 +0100
                Re: swamp rat bots Q elvis <elvis@dogonfire.com> - 2020-12-05 01:50 +0100
                Re: swamp rat bots Q grumpy@mailfence.com - 2020-12-05 02:00 +0100
    Web-bot tarpit aka spider trap (was: swamp rat bots Q) Nicolas George <george@nsup.org> - 2020-12-04 15:10 +0100
      Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Gene Heskett <gheskett@shentel.net> - 2020-12-04 17:10 +0100
        Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) <tomas@tuxteam.de> - 2020-12-04 22:10 +0100
          Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) "Martin McCormick" <martin.m@suddenlink.net> - 2020-12-06 20:20 +0100
            Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) David Christensen <dpchrist@holgerdanske.com> - 2020-12-06 21:30 +0100
            Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Charles Curley <charlescurley@charlescurley.com> - 2020-12-06 23:10 +0100
              Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Gene Heskett <gheskett@shentel.net> - 2020-12-07 02:30 +0100
                Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Doug McGarrett <dmcgarrett@optonline.net> - 2020-12-07 04:00 +0100

csiph-web