Groups > linux.debian.user > #219457

new, not nice web bots disposal

From	Gene Heskett <gheskett@shentel.net>
Newsgroups	linux.debian.user
Subject	new, not nice web bots disposal
Date	2020-02-26 10:00 +0100
Message-ID	<zE0HT-3Z7-3@gated-at.bofh.it> (permalink)
Organization	none,nada,zip

Show all headers | View raw

over the last 90 days or so, we seem to have been plauged with a new 
breed of bots scanning our web pages, and they are not just indexing our 
web pages I don't mind that, but they are ignoring our robots.txt and 
are  mirroring anything apache2 can reach, including stuff thats there 
but not reachable by a normal browser just looking around and clicking 
on links.  Its annoying as hell and when you're out in the pucker-brush 
on a 10 megabit ADSL, eats up ones available upload bandwidth of about 
275kbytes/s.  According to my cable billing, these A-H's used over 100Gb 
of my bandwidth in Nov 2019. That describes in printable language as a 
DDOS in my vocabulary.

So I asked a few questions and wrote some little 2-3 line scripts after 
putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs 
enough info you can generally identify the bots with it.

I have since have generated 49 iptables rules that have blocked 99% of 
them.

Those scripts I've placed in /etc/iptables and are owned by root.
To start iptables after a reboot:You might run this one first one 
from /etc/rc.local

root@coyote:iptables$ cat start-iptables

#!/bin/bash
cd /etc/iptables
iptables-restore <rules.v4

To add a new rule, covering that whole 256 address block because they 
seem to have a random address, changed about weekly, in that block:

root@coyote:iptables$ cat iptables-add

#!/bin/bash
iptables -I INPUT -s add.ress.to.block/24 -j DROP

Substituting the address of the offender for add.ress.to.block in the 
last tine above.

to save the rules:
root@coyote:iptables$ cat iptables-saveem

#!/bin/bash
iptables-save >rules.v4

To see what you've got so far:
root@coyote:iptables$ cat iptables-status

#!/bin/bash
iptables -L -nv --line-numbers

Which will output the rules in effect plus the hits accumulated in this 
uptime so far, in this format:
lnum   hits  bytes fate
24     846   50760 DROP  all  --  *  *   66.249.64.0/24   0.0.0.0/0

Be my guest folks, reclaim the net, we are paying for the bandwidth these 
jerks are burning up.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Back to linux.debian.user | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 10:00 +0100
  Re: new, not nice web bots disposal john doe <johndoe65534@mail.com> - 2020-02-26 10:20 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:10 +0100
      Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-26 15:30 +0100
  Re: new, not nice web bots disposal Jonas Smedegaard <jonas@jones.dk> - 2020-02-26 10:30 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:20 +0100
  Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-26 12:20 +0100
    Re: new, not nice web bots disposal Roger Price <debian@rogerprice.org> - 2020-02-26 12:50 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:30 +0100
      Re: new, not nice web bots disposal Klaus Singvogel <deb-user-ml@singvogel.net> - 2020-02-26 15:30 +0100
  Re: new, not nice web bots disposal Greg Wooledge <wooledg@eeg.ccf.org> - 2020-02-26 14:30 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:50 +0100
      Re: new, not nice web bots disposal Roger Price <debian@rogerprice.org> - 2020-02-26 16:50 +0100
        Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:10 +0100
  Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-26 20:00 +0100
    Re: new, not nice web bots disposal Reco <recoverym4n@enotuniq.net> - 2020-02-26 20:00 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:20 +0100
        Re: new, not nice web bots disposal Reco <recoverym4n@enotuniq.net> - 2020-02-26 20:30 +0100
          Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:50 +0100
      Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-26 22:10 +0100
        Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 22:50 +0100
          Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 05:30 +0100
            Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 06:10 +0100
              Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
                Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 14:30 +0100
                Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 15:20 +0100
                Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 15:40 +0100
                Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 16:00 +0100
                Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-27 16:20 +0100
                OT: LIbrarian (was: Re: new, not nice web bots disposal) rhkramer@gmail.com - 2020-02-27 16:40 +0100
            Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
              Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 16:10 +0100
                Re: new, not nice web bots disposal tomas@tuxteam.de - 2020-02-27 16:20 +0100
                Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 16:40 +0100
                Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 19:40 +0100
          Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
            Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 14:20 +0100
              Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 15:20 +0100
              Re: new, not nice web bots disposal songbird <songbird@anthive.com> - 2020-02-28 06:20 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:20 +0100
      Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-26 20:50 +0100
  Re: new, not nice web bots disposal deloptes <deloptes@gmail.com> - 2020-02-26 21:00 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 22:30 +0100

csiph-web