Groups > linux.debian.user > #219461

Re: new, not nice web bots disposal

From	Dan Ritter <dsr@randomstring.org>
Newsgroups	linux.debian.user
Subject	Re: new, not nice web bots disposal
Date	2020-02-26 12:20 +0100
Message-ID	<zE2To-6lY-7@gated-at.bofh.it> (permalink)
References	<zE0HT-3Z7-3@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

Gene Heskett wrote: 
> over the last 90 days or so, we seem to have been plauged with a new 
> breed of bots scanning our web pages, and they are not just indexing our 
> web pages I don't mind that, but they are ignoring our robots.txt and 
> are  mirroring anything apache2 can reach, including stuff thats there 
> but not reachable by a normal browser just looking around and clicking 
> on links.  Its annoying as hell and when you're out in the pucker-brush 
> on a 10 megabit ADSL, eats up ones available upload bandwidth of about 
> 275kbytes/s.  According to my cable billing, these A-H's used over 100Gb 
> of my bandwidth in Nov 2019. That describes in printable language as a 
> DDOS in my vocabulary.
> 
> So I asked a few questions and wrote some little 2-3 line scripts after 
> putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs 
> enough info you can generally identify the bots with it.
> 
> I have since have generated 49 iptables rules that have blocked 99% of 
> them.

If you find yourself needing to add lots more rules, you might
want to generate a "set" instead of individual rules:

http://ipset.netfilter.org/

https://www.linuxjournal.com/content/advanced-firewall-configurations-ipset

might be useful.

-dsr-

Thread

new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 10:00 +0100
  Re: new, not nice web bots disposal john doe <johndoe65534@mail.com> - 2020-02-26 10:20 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:10 +0100
      Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-26 15:30 +0100
  Re: new, not nice web bots disposal Jonas Smedegaard <jonas@jones.dk> - 2020-02-26 10:30 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:20 +0100
  Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-26 12:20 +0100
    Re: new, not nice web bots disposal Roger Price <debian@rogerprice.org> - 2020-02-26 12:50 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:30 +0100
      Re: new, not nice web bots disposal Klaus Singvogel <deb-user-ml@singvogel.net> - 2020-02-26 15:30 +0100
  Re: new, not nice web bots disposal Greg Wooledge <wooledg@eeg.ccf.org> - 2020-02-26 14:30 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:50 +0100
      Re: new, not nice web bots disposal Roger Price <debian@rogerprice.org> - 2020-02-26 16:50 +0100
        Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:10 +0100
  Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-26 20:00 +0100
    Re: new, not nice web bots disposal Reco <recoverym4n@enotuniq.net> - 2020-02-26 20:00 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:20 +0100
        Re: new, not nice web bots disposal Reco <recoverym4n@enotuniq.net> - 2020-02-26 20:30 +0100
          Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:50 +0100
      Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-26 22:10 +0100
        Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 22:50 +0100
          Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 05:30 +0100
            Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 06:10 +0100
              Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
                Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 14:30 +0100
                Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 15:20 +0100
                Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 15:40 +0100
                Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 16:00 +0100
                Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-27 16:20 +0100
                OT: LIbrarian (was: Re: new, not nice web bots disposal) rhkramer@gmail.com - 2020-02-27 16:40 +0100
            Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
              Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 16:10 +0100
                Re: new, not nice web bots disposal tomas@tuxteam.de - 2020-02-27 16:20 +0100
                Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 16:40 +0100
                Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 19:40 +0100
          Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
            Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 14:20 +0100
              Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 15:20 +0100
              Re: new, not nice web bots disposal songbird <songbird@anthive.com> - 2020-02-28 06:20 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:20 +0100
      Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-26 20:50 +0100
  Re: new, not nice web bots disposal deloptes <deloptes@gmail.com> - 2020-02-26 21:00 +0100
    Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 22:30 +0100

csiph-web