Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #219457 > unrolled thread

new, not nice web bots disposal

Back to article view | Back to linux.debian.user

Contents

  new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 10:00 +0100
    Re: new, not nice web bots disposal john doe <johndoe65534@mail.com> - 2020-02-26 10:20 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:10 +0100
        Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-26 15:30 +0100
    Re: new, not nice web bots disposal Jonas Smedegaard <jonas@jones.dk> - 2020-02-26 10:30 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:20 +0100
    Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-26 12:20 +0100
      Re: new, not nice web bots disposal Roger Price <debian@rogerprice.org> - 2020-02-26 12:50 +0100
        Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:30 +0100
        Re: new, not nice web bots disposal Klaus Singvogel <deb-user-ml@singvogel.net> - 2020-02-26 15:30 +0100
    Re: new, not nice web bots disposal Greg Wooledge <wooledg@eeg.ccf.org> - 2020-02-26 14:30 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 15:50 +0100
        Re: new, not nice web bots disposal Roger Price <debian@rogerprice.org> - 2020-02-26 16:50 +0100
          Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:10 +0100
    Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-26 20:00 +0100
      Re: new, not nice web bots disposal Reco <recoverym4n@enotuniq.net> - 2020-02-26 20:00 +0100
        Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:20 +0100
          Re: new, not nice web bots disposal Reco <recoverym4n@enotuniq.net> - 2020-02-26 20:30 +0100
            Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:50 +0100
        Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-26 22:10 +0100
          Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 22:50 +0100
            Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 05:30 +0100
              Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 06:10 +0100
                Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
                  Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 14:30 +0100
                    Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 15:20 +0100
                      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 15:40 +0100
                        Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 16:00 +0100
                          Re: new, not nice web bots disposal Dan Ritter <dsr@randomstring.org> - 2020-02-27 16:20 +0100
                        OT: LIbrarian (was: Re: new, not nice web bots disposal) rhkramer@gmail.com - 2020-02-27 16:40 +0100
              Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
                Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 16:10 +0100
                  Re: new, not nice web bots disposal tomas@tuxteam.de - 2020-02-27 16:20 +0100
                    Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-27 16:40 +0100
                  Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 19:40 +0100
            Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 10:00 +0100
              Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-27 14:20 +0100
                Re: new, not nice web bots disposal <tomas@tuxteam.de> - 2020-02-27 15:20 +0100
                Re: new, not nice web bots disposal songbird <songbird@anthive.com> - 2020-02-28 06:20 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 20:20 +0100
        Re: new, not nice web bots disposal Lee <ler762@gmail.com> - 2020-02-26 20:50 +0100
    Re: new, not nice web bots disposal deloptes <deloptes@gmail.com> - 2020-02-26 21:00 +0100
      Re: new, not nice web bots disposal Gene Heskett <gheskett@shentel.net> - 2020-02-26 22:30 +0100

Page 1 of 3  [1] 2 3  Next page →

#219457 — new, not nice web bots disposal

over the last 90 days or so, we seem to have been plauged with a new 
breed of bots scanning our web pages, and they are not just indexing our 
web pages I don't mind that, but they are ignoring our robots.txt and 
are  mirroring anything apache2 can reach, including stuff thats there 
but not reachable by a normal browser just looking around and clicking 
on links.  Its annoying as hell and when you're out in the pucker-brush 
on a 10 megabit ADSL, eats up ones available upload bandwidth of about 
275kbytes/s.  According to my cable billing, these A-H's used over 100Gb 
of my bandwidth in Nov 2019. That describes in printable language as a 
DDOS in my vocabulary.

So I asked a few questions and wrote some little 2-3 line scripts after 
putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs 
enough info you can generally identify the bots with it.

I have since have generated 49 iptables rules that have blocked 99% of 
them.

Those scripts I've placed in /etc/iptables and are owned by root.
To start iptables after a reboot:You might run this one first one 
from /etc/rc.local

root@coyote:iptables$ cat start-iptables

#!/bin/bash
cd /etc/iptables
iptables-restore <rules.v4

To add a new rule, covering that whole 256 address block because they 
seem to have a random address, changed about weekly, in that block:

root@coyote:iptables$ cat iptables-add

#!/bin/bash
iptables -I INPUT -s add.ress.to.block/24 -j DROP

Substituting the address of the offender for add.ress.to.block in the 
last tine above.

to save the rules:
root@coyote:iptables$ cat iptables-saveem

#!/bin/bash
iptables-save >rules.v4

To see what you've got so far:
root@coyote:iptables$ cat iptables-status

#!/bin/bash
iptables -L -nv --line-numbers

Which will output the rules in effect plus the hits accumulated in this 
uptime so far, in this format:
lnum   hits  bytes fate
24     846   50760 DROP  all  --  *  *   66.249.64.0/24   0.0.0.0/0

Be my guest folks, reclaim the net, we are paying for the bandwidth these 
jerks are burning up.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [next] | [standalone]

#219458

On 2/26/2020 9:57 AM, Gene Heskett wrote:
> over the last 90 days or so, we seem to have been plauged with a new
> breed of bots scanning our web pages, and they are not just indexing our
> web pages I don't mind that, but they are ignoring our robots.txt and
> are  mirroring anything apache2 can reach, including stuff thats there
> but not reachable by a normal browser just looking around and clicking
> on links.  Its annoying as hell and when you're out in the pucker-brush
> on a 10 megabit ADSL, eats up ones available upload bandwidth of about
> 275kbytes/s.  According to my cable billing, these A-H's used over 100Gb
> of my bandwidth in Nov 2019. That describes in printable language as a
> DDOS in my vocabulary.
>
> So I asked a few questions and wrote some little 2-3 line scripts after
> putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs
> enough info you can generally identify the bots with it.
>
> I have since have generated 49 iptables rules that have blocked 99% of
> them.
>
> Those scripts I've placed in /etc/iptables and are owned by root.
> To start iptables after a reboot:You might run this one first one
> from /etc/rc.local
>
> root@coyote:iptables$ cat start-iptables
>
> #!/bin/bash
> cd /etc/iptables
> iptables-restore <rules.v4
>
> To add a new rule, covering that whole 256 address block because they
> seem to have a random address, changed about weekly, in that block:
>
> root@coyote:iptables$ cat iptables-add
>
> #!/bin/bash
> iptables -I INPUT -s add.ress.to.block/24 -j DROP
>
> Substituting the address of the offender for add.ress.to.block in the
> last tine above.
>
> to save the rules:
> root@coyote:iptables$ cat iptables-saveem
>
> #!/bin/bash
> iptables-save >rules.v4
>
> To see what you've got so far:
> root@coyote:iptables$ cat iptables-status
>
> #!/bin/bash
> iptables -L -nv --line-numbers
>
> Which will output the rules in effect plus the hits accumulated in this
> uptime so far, in this format:
> lnum   hits  bytes fate
> 24     846   50760 DROP  all  --  *  *   66.249.64.0/24   0.0.0.0/0
>
> Be my guest folks, reclaim the net, we are paying for the bandwidth these
> jerks are burning up.
>

The above is the way the OP has choosen to go about it but configuring
apache properly using fail2ban in addition of the robot.txt file. should
also be considered

See past threads from this OP for an history of this.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#219473

On Wednesday 26 February 2020 04:05:53 john doe wrote:

> On 2/26/2020 9:57 AM, Gene Heskett wrote:
> > over the last 90 days or so, we seem to have been plauged with a new
> > breed of bots scanning our web pages, and they are not just indexing
> > our web pages I don't mind that, but they are ignoring our
> > robots.txt and are  mirroring anything apache2 can reach, including
> > stuff thats there but not reachable by a normal browser just looking
> > around and clicking on links.  Its annoying as hell and when you're
> > out in the pucker-brush on a 10 megabit ADSL, eats up ones available
> > upload bandwidth of about 275kbytes/s.  According to my cable
> > billing, these A-H's used over 100Gb of my bandwidth in Nov 2019.
> > That describes in printable language as a DDOS in my vocabulary.
> >
> > So I asked a few questions and wrote some little 2-3 line scripts
> > after putting a tail on /var/lib/httpd/other_vhosts_access.log,
> > which logs enough info you can generally identify the bots with it.
> >
> > I have since have generated 49 iptables rules that have blocked 99%
> > of them.
> >
> > Those scripts I've placed in /etc/iptables and are owned by root.
> > To start iptables after a reboot:You might run this one first one
> > from /etc/rc.local
> >
> > root@coyote:iptables$ cat start-iptables
> >
> > #!/bin/bash
> > cd /etc/iptables
> > iptables-restore <rules.v4
> >
> > To add a new rule, covering that whole 256 address block because
> > they seem to have a random address, changed about weekly, in that
> > block:
> >
> > root@coyote:iptables$ cat iptables-add
> >
> > #!/bin/bash
> > iptables -I INPUT -s add.ress.to.block/24 -j DROP
> >
> > Substituting the address of the offender for add.ress.to.block in
> > the last tine above.
> >
> > to save the rules:
> > root@coyote:iptables$ cat iptables-saveem
> >
> > #!/bin/bash
> > iptables-save >rules.v4
> >
> > To see what you've got so far:
> > root@coyote:iptables$ cat iptables-status
> >
> > #!/bin/bash
> > iptables -L -nv --line-numbers
> >
> > Which will output the rules in effect plus the hits accumulated in
> > this uptime so far, in this format:
> > lnum   hits  bytes fate
> > 24     846   50760 DROP  all  --  *  *   66.249.64.0/24   0.0.0.0/0
> >
> > Be my guest folks, reclaim the net, we are paying for the bandwidth
> > these jerks are burning up.
>
> The above is the way the OP has choosen to go about it but configuring
> apache properly using fail2ban in addition of the robot.txt file.
> should also be considered

That was also suggested and tried, for about a week, but there were no 
failures to initiate the ban from. So I got out a bigger hammer.  This 
works for us little folks. If those bots want to be nice and just index 
my pages, I'd not have a problem as it would help advertise that I might 
have something they'd want. But they want to mirror the whole site, then 
each one wants to do it again half a week later, before they've got the 
first full copy, and theres several hundred of them doing it. That is a 
DDOS when they burn up all my upload bandwidth on a 24/7 basis. As an 
old friend of mine, JoAnne Dow has said, trying to be a lady, in this 
case years before Linux was written, screewwuum.  We were still useing 
Amiga's and a web browser called miami in those days, before that pots 
and a 300 baud modem making calls from a trs-80 color computer 1 with a 
grand total of 64k for memory.

We were talking about a WD hard drive whose firmware could not handle a 
bigger than one 256 byte sector write at a time. ISTR it was a 15 
megabyte drive, then currently state of the art. WD's response? Works on 
dos, shrug.

What I fail to understand is why the universal reticence on the part of 
all the knowitalls here to use the almost 20 year old tools given us to 
control these jerks? Particularly when it looks like its the only tool 
that actually and effectively works.

> See past threads from this OP for an history of this.
>
> --
> John Doe

Who is John Doe, and how fast was your first modem, mine was 300 baud. 
And it was a long distance call 18 miles to a delphi access point. Long 
before Judge Green.

Cheers, Gene Heskett, who is not hiding behind an alias.
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219476

Gene Heskett wrote: 
> On Wednesday 26 February 2020 04:05:53 john doe wrote:
> 
> > On 2/26/2020 9:57 AM, Gene Heskett wrote:
> > > Be my guest folks, reclaim the net, we are paying for the bandwidth
> > > these jerks are burning up.
> >
> > The above is the way the OP has choosen to go about it but configuring
> > apache properly using fail2ban in addition of the robot.txt file.
> > should also be considered
> 
> That was also suggested and tried, for about a week, but there were no 
> failures to initiate the ban from. So I got out a bigger hammer.  This 

While fail2ban comes with a bunch of recognizers, it's fairly
easy to customize it -- anything that generates a log file with
an IP address and some condition that you can pick up with a
regex is fair game.

> What I fail to understand is why the universal reticence on the part of 
> all the knowitalls here to use the almost 20 year old tools given us to 
> control these jerks? Particularly when it looks like its the only tool 
> that actually and effectively works.

In this case of this knowitall, it's because I've used those
tools, written my own -- just like you -- and I have come to
appreciate the value of sharing tools -- just like you -- so
when I see that lots of people recommend a tool, I find some
value in seeing if it's right for me.

If someone went to the bother of packaging it for Debian, and
other people say it's a good tool -- maybe it is. And maybe 
they are wrong. And it could certainly not be the right tool
for me, or for you, in this particular circumstance.

-dsr-

[toc] | [prev] | [next] | [standalone]

#219459

[Multipart message — attachments visible in raw view] — view raw

Quoting Gene Heskett (2020-02-26 09:57:51)
> over the last 90 days or so, we seem to have been plauged with a new 
> breed of bots scanning our web pages, and they are not just indexing 
> our web pages I don't mind that, but they are ignoring our robots.txt 
> and are mirroring anything apache2 can reach, including stuff thats 
> there but not reachable by a normal browser just looking around and 
> clicking on links.  Its annoying as hell and when you're out in the 
> pucker-brush on a 10 megabit ADSL, eats up ones available upload 
> bandwidth of about 275kbytes/s.

Download "eating" upload on ADSL might be due to bufferbloat: 
https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

[toc] | [prev] | [next] | [standalone]

#219474

On Wednesday 26 February 2020 04:21:09 Jonas Smedegaard wrote:

> Quoting Gene Heskett (2020-02-26 09:57:51)
>
> > over the last 90 days or so, we seem to have been plauged with a new
> > breed of bots scanning our web pages, and they are not just indexing
> > our web pages I don't mind that, but they are ignoring our
> > robots.txt and are mirroring anything apache2 can reach, including
> > stuff thats there but not reachable by a normal browser just looking
> > around and clicking on links.  Its annoying as hell and when you're
> > out in the pucker-brush on a 10 megabit ADSL, eats up ones available
> > upload bandwidth of about 275kbytes/s.
>
> Download "eating" upload on ADSL might be due to bufferbloat:
> https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bu
>fferbloat/
>
Maybe, but I can still go read the news IF my browser can sneak a page 
request in between upload packets when gkrellm is showing 300k of upload 
at the instant.
>
And I'm subscribed Jonas, no need to reply all.


>  - Jonas


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219461

Gene Heskett wrote: 
> over the last 90 days or so, we seem to have been plauged with a new 
> breed of bots scanning our web pages, and they are not just indexing our 
> web pages I don't mind that, but they are ignoring our robots.txt and 
> are  mirroring anything apache2 can reach, including stuff thats there 
> but not reachable by a normal browser just looking around and clicking 
> on links.  Its annoying as hell and when you're out in the pucker-brush 
> on a 10 megabit ADSL, eats up ones available upload bandwidth of about 
> 275kbytes/s.  According to my cable billing, these A-H's used over 100Gb 
> of my bandwidth in Nov 2019. That describes in printable language as a 
> DDOS in my vocabulary.
> 
> So I asked a few questions and wrote some little 2-3 line scripts after 
> putting a tail on /var/lib/httpd/other_vhosts_access.log, which logs 
> enough info you can generally identify the bots with it.
> 
> I have since have generated 49 iptables rules that have blocked 99% of 
> them.

If you find yourself needing to add lots more rules, you might
want to generate a "set" instead of individual rules:

http://ipset.netfilter.org/

https://www.linuxjournal.com/content/advanced-firewall-configurations-ipset

might be useful.

-dsr-

[toc] | [prev] | [next] | [standalone]

#219462

On Wed, 26 Feb 2020, Dan Ritter wrote:

> If you find yourself needing to add lots more rules, you might want to 
> generate a "set" instead of individual rules:
>
> http://ipset.netfilter.org/
> https://www.linuxjournal.com/content/advanced-firewall-configurations-ipset
> might be useful.

I find ipsets the natural way of setting up rules.  I run a script which blocks 
whole countries, taking the country data from 
http://ipverse.net/ipblocks/data/countries/

Simple and efficient.  I once had a set with 140000 (yes, 140 thousand) ipblocks 
in an ipset with no apparent performance hit.

Roger

[toc] | [prev] | [next] | [standalone]

#219475

On Wednesday 26 February 2020 06:40:39 Roger Price wrote:

> On Wed, 26 Feb 2020, Dan Ritter wrote:
> > If you find yourself needing to add lots more rules, you might want
> > to generate a "set" instead of individual rules:
> >
> > http://ipset.netfilter.org/
> > https://www.linuxjournal.com/content/advanced-firewall-configuration
> >s-ipset might be useful.
>
> I find ipsets the natural way of setting up rules.  I run a script
> which blocks whole countries, taking the country data from
> http://ipverse.net/ipblocks/data/countries/
>
> Simple and efficient.  I once had a set with 140000 (yes, 140
> thousand) ipblocks in an ipset with no apparent performance hit.
>
> Roger
Roger, and anyone else following along, I am blocking 12544 with 49 
rules. based on 49 rules, each blocking /24 addresses.  Seems to be 
sufficient.


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219477

Roger Price wrote:
> I find ipsets the natural way of setting up rules.  I run a script which
> blocks whole countries, taking the country data from
> http://ipverse.net/ipblocks/data/countries/

Not a bad idea, but the database is sometimes wrong. Examples:

Duplicates (shall not be possible, but there exist entries):
- 194.114.128.0 is listed in ru.zone and se.zone
- 151.0.0.0 is listed in it.zone and ee.zone

Wrong entries:
- 109.70.64.0 is listed in de.zone, but should be in nl.zone
- 116.202.0.0 is listed in de.zone, but should be in zz.zone (unlocated)

Made only a first, rough test of my personal entries.

Be cautious when playing with such things. :-)

Best regards,
	Klaus.
-- 
Klaus Singvogel
GnuPG-Key-ID: 1024R/5068792D  1994-06-27

[toc] | [prev] | [next] | [standalone]

#219468

On Wed, Feb 26, 2020 at 03:57:51AM -0500, Gene Heskett wrote:
> over the last 90 days or so, we seem to have been plauged with a new 
> breed of bots scanning our web pages, and they are not just indexing our 
> web pages I don't mind that, but they are ignoring our robots.txt

I can believe this.  But it's better if you SHOW us the actual logfile
entries (not all of them, obviously -- just a reasonable sample).

If the bots have a recognizable user-agent, you can just configure
your web server to block queries from that user-agent.  Then you don't
have to deal with iptables, nftables, or whatever they're calling
it this year.

[toc] | [prev] | [next] | [standalone]

#219478

On Wednesday 26 February 2020 08:27:33 Greg Wooledge wrote:

> On Wed, Feb 26, 2020 at 03:57:51AM -0500, Gene Heskett wrote:
> > over the last 90 days or so, we seem to have been plauged with a new
> > breed of bots scanning our web pages, and they are not just indexing
> > our web pages I don't mind that, but they are ignoring our
> > robots.txt
>
> I can believe this.  But it's better if you SHOW us the actual logfile
> entries (not all of them, obviously -- just a reasonable sample).
>
> If the bots have a recognizable user-agent, you can just configure
> your web server to block queries from that user-agent.  Then you don't
> have to deal with iptables, nftables, or whatever they're calling
> it this year.

I've had this in place now long enough the good examples like all the 
russian sembot's have probably all logrotated into /dev/null. This just 
showed up this morning, but no clue what it might be.  Blocked it 
anyway. 46th rule.

coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:06:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:06:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:07:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:07:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:08:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:08:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:09:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:09:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:10:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:10:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:11:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:11:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:12:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:12:58 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:13:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:13:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:14:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:14:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:15:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:15:59 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:51:43 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:51:43 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:20:29:32 -0500] "-" 
408 0 "-" "-"
coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:20:29:32 -0500] "-" 
408 0 "-" "-"

No clue, butt dial? PROXAD.net someplace in France.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219482

On Wed, 26 Feb 2020, Gene Heskett wrote:

> This just showed up this morning, but no clue what it might be.  Blocked it 
> anyway. 46th rule.
>
> coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:06:58 -0500] "-"
> 408 0 "-" "-"
>
> No clue, butt dial? PROXAD.net someplace in France.

That's my ISP.  It's called "Free".  It features some real idiots who have to 
be blocked.

Roger

[toc] | [prev] | [next] | [standalone]

#219490

On Wednesday 26 February 2020 10:43:13 Roger Price wrote:

> On Wed, 26 Feb 2020, Gene Heskett wrote:
> > This just showed up this morning, but no clue what it might be. 
> > Blocked it anyway. 46th rule.
> >
> > coyote.coyote.den:80 91.160.218.196 - - [25/Feb/2020:19:06:58 -0500]
> > "-" 408 0 "-" "-"
> >
> > No clue, butt dial? PROXAD.net someplace in France.
>
> That's my ISP.  It's called "Free".  It features some real idiots who
> have to be blocked.
>
> Roger

And that, from the whois listing, will take a /16 to block, and I do find 
that distastefull. Too many innocents would be nailed to the wall. So 
I'll just block that /24 as needed.

To all, give me an email, or a shout here if you find the link in my sig 
doesn't work.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219488

On 2/26/20, Gene Heskett <gheskett@shentel.net> wrote:
> over the last 90 days or so, we seem to have been plauged with a new
> breed of bots scanning our web pages, and they are not just indexing our
> web pages I don't mind that, but they are ignoring our robots.txt and
> are  mirroring anything apache2 can reach, including stuff thats there
> but not reachable by a normal browser just looking around and clicking
> on links.
  <.. snip ..>
> To add a new rule, covering that whole 256 address block because they
> seem to have a random address, changed about weekly, in that block:
>
> root@coyote:iptables$ cat iptables-add
>
> #!/bin/bash
> iptables -I INPUT -s add.ress.to.block/24 -j DROP

Have you considered REJECT instead of DROP?

REJECT should send a RST telling the other side to give up now.
DROP just drops the packet leaving the other side to retry until the
retry limit is hit.

Lee

[toc] | [prev] | [next] | [standalone]

#219489

	Hi.

On Wed, Feb 26, 2020 at 01:50:40PM -0500, Lee wrote:
> On 2/26/20, Gene Heskett <gheskett@shentel.net> wrote:
> > over the last 90 days or so, we seem to have been plauged with a new
> > breed of bots scanning our web pages, and they are not just indexing our
> > web pages I don't mind that, but they are ignoring our robots.txt and
> > are  mirroring anything apache2 can reach, including stuff thats there
> > but not reachable by a normal browser just looking around and clicking
> > on links.
>   <.. snip ..>
> > To add a new rule, covering that whole 256 address block because they
> > seem to have a random address, changed about weekly, in that block:
> >
> > root@coyote:iptables$ cat iptables-add
> >
> > #!/bin/bash
> > iptables -I INPUT -s add.ress.to.block/24 -j DROP
> 
> Have you considered REJECT instead of DROP?

A neat idea for your LAN. A bad idea in this case.

You *want* that other side to retry, wasting their time instead of
spamming their target. In fact, one should consider using TARPIT instead
of a DROP here.

Reco

[toc] | [prev] | [next] | [standalone]

#219494

On Wednesday 26 February 2020 13:54:09 Reco wrote:

> 	Hi.
>
> On Wed, Feb 26, 2020 at 01:50:40PM -0500, Lee wrote:
> > On 2/26/20, Gene Heskett <gheskett@shentel.net> wrote:
> > > over the last 90 days or so, we seem to have been plauged with a
> > > new breed of bots scanning our web pages, and they are not just
> > > indexing our web pages I don't mind that, but they are ignoring
> > > our robots.txt and are  mirroring anything apache2 can reach,
> > > including stuff thats there but not reachable by a normal browser
> > > just looking around and clicking on links.
> >
> >   <.. snip ..>
> >
> > > To add a new rule, covering that whole 256 address block because
> > > they seem to have a random address, changed about weekly, in that
> > > block:
> > >
> > > root@coyote:iptables$ cat iptables-add
> > >
> > > #!/bin/bash
> > > iptables -I INPUT -s add.ress.to.block/24 -j DROP
> >
> > Have you considered REJECT instead of DROP?
>
> A neat idea for your LAN. A bad idea in this case.
>
> You *want* that other side to retry, wasting their time instead of
> spamming their target. In fact, one should consider using TARPIT
> instead of a DROP here.
>
> Reco

Now thats a thought, does it take much fiddling to set that up?  Links to 
tuts plz.

Thanks Reco

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219495

On Wed, Feb 26, 2020 at 02:15:18PM -0500, Gene Heskett wrote:
> On Wednesday 26 February 2020 13:54:09 Reco wrote:
> 
> > 	Hi.
> >
> > On Wed, Feb 26, 2020 at 01:50:40PM -0500, Lee wrote:
> > > On 2/26/20, Gene Heskett <gheskett@shentel.net> wrote:
> > > > over the last 90 days or so, we seem to have been plauged with a
> > > > new breed of bots scanning our web pages, and they are not just
> > > > indexing our web pages I don't mind that, but they are ignoring
> > > > our robots.txt and are  mirroring anything apache2 can reach,
> > > > including stuff thats there but not reachable by a normal browser
> > > > just looking around and clicking on links.
> > >
> > >   <.. snip ..>
> > >
> > > > To add a new rule, covering that whole 256 address block because
> > > > they seem to have a random address, changed about weekly, in that
> > > > block:
> > > >
> > > > root@coyote:iptables$ cat iptables-add
> > > >
> > > > #!/bin/bash
> > > > iptables -I INPUT -s add.ress.to.block/24 -j DROP
> > >
> > > Have you considered REJECT instead of DROP?
> >
> > A neat idea for your LAN. A bad idea in this case.
> >
> > You *want* that other side to retry, wasting their time instead of
> > spamming their target. In fact, one should consider using TARPIT
> > instead of a DROP here.
> 
> Now thats a thought, does it take much fiddling to set that up?  Links to 
> tuts plz.

You have to use testing or sid for that. [1] is a place to start,
[2] will be needed too.

[1] https://packages.debian.org/bullseye/xtables-addons-dkms
[2] https://packages.debian.org/bullseye/xtables-addons-common

Reco

[toc] | [prev] | [next] | [standalone]

#219496

On Wednesday 26 February 2020 14:21:31 Reco wrote:

> On Wed, Feb 26, 2020 at 02:15:18PM -0500, Gene Heskett wrote:
> > On Wednesday 26 February 2020 13:54:09 Reco wrote:
> > > 	Hi.
> > >
> > > On Wed, Feb 26, 2020 at 01:50:40PM -0500, Lee wrote:
> > > > On 2/26/20, Gene Heskett <gheskett@shentel.net> wrote:
> > > > > over the last 90 days or so, we seem to have been plauged with
> > > > > a new breed of bots scanning our web pages, and they are not
> > > > > just indexing our web pages I don't mind that, but they are
> > > > > ignoring our robots.txt and are  mirroring anything apache2
> > > > > can reach, including stuff thats there but not reachable by a
> > > > > normal browser just looking around and clicking on links.
> > > >
> > > >   <.. snip ..>
> > > >
> > > > > To add a new rule, covering that whole 256 address block
> > > > > because they seem to have a random address, changed about
> > > > > weekly, in that block:
> > > > >
> > > > > root@coyote:iptables$ cat iptables-add
> > > > >
> > > > > #!/bin/bash
> > > > > iptables -I INPUT -s add.ress.to.block/24 -j DROP
> > > >
> > > > Have you considered REJECT instead of DROP?
> > >
> > > A neat idea for your LAN. A bad idea in this case.
> > >
> > > You *want* that other side to retry, wasting their time instead of
> > > spamming their target. In fact, one should consider using TARPIT
> > > instead of a DROP here.
> >
> > Now thats a thought, does it take much fiddling to set that up? 
> > Links to tuts plz.
>
> You have to use testing or sid for that. [1] is a place to start,
> [2] will be needed too.
>
> [1] https://packages.debian.org/bullseye/xtables-addons-dkms
> [2] https://packages.debian.org/bullseye/xtables-addons-common
>
> Reco

Ahh, and I'm still on stretch, 9-11 so I'll have to start by upgrading to 
buster.  Is there a recipe for that, its about time and I am beginning 
to like what I see of buster on an rpi4b. I've been dragging my feet 
waiting for the linuxcnc guys to spin a buster install iso, but lots of 
python stuff has disappeared, making it very difficult to make all its 
toys work on buster. If my results of building it on buster are any 
indication, there's likely 20+ python libraries we'll have to assume 
support for just for x86 stuffs.

My machines that are running machines don't have near as much 
gingerbready stuff installed, so they will be easier than this one to 
update.  And the rest are still on wheezy. But I'm not too worried, I've 
got dd-wrt between me and the black hat crowd.

All the other tarpit thingies I've looked at take more work than I've 
time to play setting them up.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#219504

[Multipart message — attachments visible in raw view] — view raw

On Wed, Feb 26, 2020 at 09:54:09PM +0300, Reco wrote:
> 	Hi.
> 
> On Wed, Feb 26, 2020 at 01:50:40PM -0500, Lee wrote:

[...]

> > Have you considered REJECT instead of DROP?
> 
> A neat idea for your LAN. A bad idea in this case.

Exactly.

> You *want* that other side to retry, wasting their time instead of
> spamming their target. In fact, one should consider using TARPIT instead
> of a DROP here.

Moreover: you don't want the other side to even know that you're
there. The less info you give away the better.

In a LAN, however, REJECT is far better: you want the other side
to know that you're there, but not talking.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web