Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #229244 > unrolled thread

swamp rat bots Q

Back to article view | Back to linux.debian.user

Contents

  swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 14:00 +0100
    Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-03 14:10 +0100
      Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 15:10 +0100
        Re: swamp rat bots Q Greg Wooledge <wooledg@eeg.ccf.org> - 2020-12-03 15:20 +0100
          Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-03 20:00 +0100
            Re: swamp rat bots Q Håkon Alstadheim <hakon@alstadheim.priv.no> - 2020-12-03 22:00 +0100
            Re: swamp rat bots Q Keith Christian <keith1christian@gmail.com> - 2020-12-03 22:30 +0100
        Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-03 16:40 +0100
    Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 03:20 +0100
      Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 06:10 +0100
        Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 07:10 +0100
          Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-04 07:40 +0100
            Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 08:00 +0100
          Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 07:50 +0100
            Re: swamp rat bots Q john doe <johndoe65534@mail.com> - 2020-12-04 08:20 +0100
            Re: swamp rat bots Q Andy Smith <andy@strugglers.net> - 2020-12-04 08:50 +0100
        Re: swamp rat bots Q "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2020-12-04 11:50 +0100
          Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 14:50 +0100
        Re: swamp rat bots Q Andrei POPESCU <andreimpopescu@gmail.com> - 2020-12-04 12:00 +0100
    Re: swamp rat bots Q "hdv@gmail" <hdv.jadev@gmail.com> - 2020-12-04 10:00 +0100
      Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 14:50 +0100
        Re: swamp rat bots Q Reco <recoverym4n@enotuniq.net> - 2020-12-04 18:40 +0100
          Re: swamp rat bots Q grumpy@mailfence.com - 2020-12-04 19:10 +0100
            Re: swamp rat bots Q Reco <recoverym4n@enotuniq.net> - 2020-12-04 19:20 +0100
              Re: swamp rat bots Q Carl Fink <carlf@panix.com> - 2020-12-04 19:30 +0100
              Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 21:00 +0100
          Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 21:00 +0100
            Re: swamp rat bots Q Tixy <tixy@yxit.co.uk> - 2020-12-04 22:20 +0100
              Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-04 23:10 +0100
                Re: swamp rat bots Q Tixy <tixy@yxit.co.uk> - 2020-12-04 23:40 +0100
                  Re: swamp rat bots Q Gene Heskett <gheskett@shentel.net> - 2020-12-05 01:00 +0100
                    Re: swamp rat bots Q Andrei POPESCU <andreimpopescu@gmail.com> - 2020-12-05 11:00 +0100
                  Re: swamp rat bots Q elvis <elvis@dogonfire.com> - 2020-12-05 01:50 +0100
                    Re: swamp rat bots Q grumpy@mailfence.com - 2020-12-05 02:00 +0100
      Web-bot tarpit aka spider trap (was: swamp rat bots Q) Nicolas George <george@nsup.org> - 2020-12-04 15:10 +0100
        Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Gene Heskett <gheskett@shentel.net> - 2020-12-04 17:10 +0100
          Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) <tomas@tuxteam.de> - 2020-12-04 22:10 +0100
            Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) "Martin McCormick" <martin.m@suddenlink.net> - 2020-12-06 20:20 +0100
              Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) David Christensen <dpchrist@holgerdanske.com> - 2020-12-06 21:30 +0100
              Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Charles Curley <charlescurley@charlescurley.com> - 2020-12-06 23:10 +0100
                Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Gene Heskett <gheskett@shentel.net> - 2020-12-07 02:30 +0100
                  Re: Web-bot tarpit aka spider trap (was: swamp rat bots Q) Doug McGarrett <dmcgarrett@optonline.net> - 2020-12-07 04:00 +0100

Page 1 of 3  [1] 2 3  Next page →

#229244 — swamp rat bots Q

I've had it with a certain bot that that ignore my robots.txt and 
proceeds to mirror my site, several times a day, burning up my upload 
bandwidth. They've moved it to 5 different addresses since midnight.

I want to nail the door shut on the first attempted access by these AH's.

Does anyone have a ready made script that can watch my httpd "other" log, 
and if a certain name is at the end of the line, grabs the ipv4 src 
address as arg3 of the line, and applies it to iptables DROP rules?

Or do I have to invent a new wheel for this?

Basic rules that simplify it somewhat.

1. this is ipv4 only country and not likely to change in the future 
decade.

2. the list of offending bot names will probably never go beyond 50, if 
that many. 5 would be realistic.

3. the src address in the log is at a fixed offset, obtainable with the 
bash MID$ but the dns return will need some acrobatics involving the 
bash RIGHT$ function.

4. it should track the number of hits, and after so many in a /24 block, 
autoswitch to a /16 block in order to keep the rules file from 
exploding.

Any help will be much appreciated. PM's in this case welcome as I can't 
see broadcasting our armament against these MF'ers being broadcast on a 
public list.

Thanks all.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [next] | [standalone]

#229245

On 12/3/2020 1:35 PM, Gene Heskett wrote:
> I've had it with a certain bot that that ignore my robots.txt and
> proceeds to mirror my site, several times a day, burning up my upload
> bandwidth. They've moved it to 5 different addresses since midnight.
>
> I want to nail the door shut on the first attempted access by these AH's.
>
> Does anyone have a ready made script that can watch my httpd "other" log,
> and if a certain name is at the end of the line, grabs the ipv4 src
> address as arg3 of the line, and applies it to iptables DROP rules?
>
> Or do I have to invent a new wheel for this?
>
> Basic rules that simplify it somewhat.
>
> 1. this is ipv4 only country and not likely to change in the future
> decade.
>
> 2. the list of offending bot names will probably never go beyond 50, if
> that many. 5 would be realistic.
>
> 3. the src address in the log is at a fixed offset, obtainable with the
> bash MID$ but the dns return will need some acrobatics involving the
> bash RIGHT$ function.
>
> 4. it should track the number of hits, and after so many in a /24 block,
> autoswitch to a /16 block in order to keep the rules file from
> exploding.
>

Is that not the same question you asked a while back, I then suggested
'fail2ban' or using ip/nftables own capabilities?

It looks to me like you are making your life way much harder than it
should be.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#229250

On Thursday 03 December 2020 08:07:33 john doe wrote:

> On 12/3/2020 1:35 PM, Gene Heskett wrote:
> > I've had it with a certain bot that that ignore my robots.txt and
> > proceeds to mirror my site, several times a day, burning up my
> > upload bandwidth. They've moved it to 5 different addresses since
> > midnight.
> >
> > I want to nail the door shut on the first attempted access by these
> > AH's.
> >
> > Does anyone have a ready made script that can watch my httpd "other"
> > log, and if a certain name is at the end of the line, grabs the ipv4
> > src address as arg3 of the line, and applies it to iptables DROP
> > rules?
> >
> > Or do I have to invent a new wheel for this?
> >
> > Basic rules that simplify it somewhat.
> >
> > 1. this is ipv4 only country and not likely to change in the future
> > decade.
> >
> > 2. the list of offending bot names will probably never go beyond 50,
> > if that many. 5 would be realistic.
> >
> > 3. the src address in the log is at a fixed offset, obtainable with
> > the bash MID$ but the dns return will need some acrobatics involving
> > the bash RIGHT$ function.
> >
> > 4. it should track the number of hits, and after so many in a /24
> > block, autoswitch to a /16 block in order to keep the rules file
> > from exploding.
>
> Is that not the same question you asked a while back, I then suggested
> 'fail2ban' or using ip/nftables own capabilities?
>
Yes John. But explain to me what fail2ban is sopposed to do?

Its running, but has failed to ban anything no matter what sort of 403's 
I return.

Fail2ban has been running here for years, and in just sits there doing 
nothing, so if its as great a swiss army knife as others claim it to be, 
lets either make it work, or quit recommending it. 

I need something I can feed with a tee off the tail output, detect that 
it is one of the offending bots by name, and if so, apply its ipv4 
address to an iptables DROP rule.

> It looks to me like you are making your life way much harder than it
> should be.

Fine, now show me how to make fail2ban do something usefull. I just 
rebooted because of a drive failure and found it couldn't be found 
running by htop, so I started it. Now make it do something usefull if 
its so great.

iptables does work but I have to manually pick the addresses out of the 
log in order to put them into the rules.

They move these bots around 2 or more times a month to get around people 
like me who do use something like iptables, so Ideally I should nuke the 
rules file about monthly and restart a new compilation by feeding this 
script with the last 5000 lines of the "other" log.  And leave the tail 
active to feed new hits into this script one line at a time as they 
occur.

Thank you.
> --
> John Doe


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#229251

On Thu, Dec 03, 2020 at 09:02:47AM -0500, Gene Heskett wrote:
> Yes John. But explain to me what fail2ban is sopposed to do?

It's supposed to "monitor" (tail -F equivalent) your log files, and
look for anomalies.  If it finds one, it's supposed to take action,
which is typically adding an entry to iptables.

> Its running, but has failed to ban anything no matter what sort of 403's 
> I return.

You need to configure it.  Tell it what log files to read, what is to
be considered an anomaly, and what action to take.

[toc] | [prev] | [next] | [standalone]

#229257

On Thursday 03 December 2020 09:14:29 Greg Wooledge wrote:

> On Thu, Dec 03, 2020 at 09:02:47AM -0500, Gene Heskett wrote:
> > Yes John. But explain to me what fail2ban is sopposed to do?
>
> It's supposed to "monitor" (tail -F equivalent) your log files, and
> look for anomalies.  If it finds one, it's supposed to take action,
> which is typically adding an entry to iptables.
>
> > Its running, but has failed to ban anything no matter what sort of
> > 403's I return.
>
> You need to configure it.  Tell it what log files to read, what is to
> be considered an anomaly, and what action to take.

And where do I do that?

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#229260

Den 03.12.2020 19:58, skrev Gene Heskett:
> On Thursday 03 December 2020 09:14:29 Greg Wooledge wrote:
>
>> On Thu, Dec 03, 2020 at 09:02:47AM -0500, Gene Heskett wrote:
>>> Yes John. But explain to me what fail2ban is sopposed to do?
>> It's supposed to "monitor" (tail -F equivalent) your log files, and
>> look for anomalies.  If it finds one, it's supposed to take action,
>> which is typically adding an entry to iptables.
>>
>>> Its running, but has failed to ban anything no matter what sort of
>>> 403's I return.
>> You need to configure it.  Tell it what log files to read, what is to
>> be considered an anomaly, and what action to take.
> And where do I do that?
>
> Cheers, Gene Heskett

If you don't know where to start with a package there are two options: 
Look at the files installed, or search the net. e.g.:

<code>

$ dpkg -L fail2ban | grep man1

/usr/share/man/man1
/usr/share/man/man1/fail2ban-client.1.gz
/usr/share/man/man1/fail2ban-regex.1.gz
/usr/share/man/man1/fail2ban-server.1.gz
/usr/share/man/man1/fail2ban-testcases.1.gz
/usr/share/man/man1/fail2ban.1.gz
</code>

Obvious choice here is the eponymous man-page fail2ban. So start there. 
Once you have scanned that short page, you will be pointed to jail(5), 
i.e. the manual page on configuration. (Config-files are in section 5 of 
the man-pages. )

For a look at the filters installed but not activated, do

<code>

dpkg -L fail2ban | grep filter.d

</code>

There are interesting filters there for your case. You will probably 
want to adjust the filters for your particular nemesis. This is quite 
quick with the aid of fail2ban-regex(1) . You will probably NOT need to 
adjust which log-files are scanned, as debian installs default patterns 
for various logs-file-names. See /etc/fail2ban/paths* .


Lastly, once you have a jail with filters you like configured, make sure 
the server is running, and watch

[toc] | [prev] | [next] | [standalone]

#229263

Gene,

Fail2ban can be difficult to comprehend at first, so here are some ideas:

As either the fail2ban user (may need root), run this command to see
that fail2ban is active and what "jails" are active, a jail
corresponds to one type of message in the log file fail2ban is
watching, which are set up in /etc/fail2ban/filter.d/*.conf files and
configs in /etc/fail2ban/jail.conf:)
# fail2ban-client status

Also, look at fail2ban-client's options:
# fail2ban-client help

Fail2ban is essentially a log file miner ("tailer" as Greg said above)
that watches for the frequency that certain regex patterns appear in
the log file. The log file must have timestamps and FQDN's or
hostnames or IP addresses so that the incoming host can be identified
and the frequency of the incoming connections can be derived from the
timestamps.

Look in the /etc/fail2ban/filter.d/ directory for the files containing
regular expressions that fail2ban should use to match lines in the log
file it should monitor, try "cat
/etc/fail2ban/filter.d/apache-200.conf" to see the regex.

The two main file sets to get started with are the
/etc/fail2ban/filter.d/ files and the /etc/fail2ban/jail.conf file.
In the /etc/fail2ban/jail.conf file you'll see the filenames of the
conf files between brackets:

e.g. [apache-200] corresponds to /etc/fail2ban/filter.d/apache-200.conf

which contains settings for watching 200 OK in apache log files, and
the number of hits and the time window before the incoming host is
blocked.

Take a look at the fail2ban.log file for the latest "news:"
As root:
# tail -f /var/log/fail2ban.log
or,
# cat /var/log/fail2ban.log | tail to see what is going on, if anything.

To check whether fail2ban is running, the command below should return
a "fail2ban-server" line in the output.
# pgrep -fl fail2ban

TESTING / DEBUGGING:
Use the fail2ban-regex command to test log file samples and whether
your entries in /etc/fail2ban/filter.d/*.conf files and configs in
/etc/fail2ban/jail.conf are working without waiting around for another
inbound event.  fail2ban-regex will help you debug and/or fine-tune
the regex and timing so that fail2ban can do its job.

If the regex in the /etc/fail2ban/filter.d/*.conf file does not match
any lines in the log file that fail2ban is watching, NOTHING WILL BE
BANNED, since fail2ban does not see the timestamps and FQDN's or
hostnames or IP addresses in order to count hits and frequencies.
Hence: the regex in the /etc/fail2ban/filter.d/*.conf file is CRITICAL
to the proper operation of fail2ban.

There are a lot of conf files with regexes in
/etc/fail2ban/filter.d/*.conf so take a look for ideas.

Test with fail2ban-regex and when it shows a match, it will work in
"production."  Trim off a few lines in the log file and test like
this:
First, copy some lines from a log file such as
access_log.2020-12-03-00_00_00 to the /tmp directory and copy the
active apache.conf file in /etc/filter.d to /tmp also, so that you can
tune the regex without affecting "production."
# cp -pv /etc/fail2ban/filter.d/apache-200.conf /tmp

# cp -pv access_log.2020-12-03-00_00_00 /tmp   ## Edit the
/tmp/access_log.2020-12-03-00_00_00 file down to a few hundred lines
to speed up the debugging/tuning process.

Then, run this command line to test:
# fail2ban-regex --print-all-matched
/tmp/access_log.2020-12-03-00_00_00 /tmp/apache-200.conf

See what this produces.

That should help you get started.

Keith

[toc] | [prev] | [next] | [standalone]

#229255

On 12/3/2020 3:02 PM, Gene Heskett wrote:
> On Thursday 03 December 2020 08:07:33 john doe wrote:
>
>> On 12/3/2020 1:35 PM, Gene Heskett wrote:
>>> I've had it with a certain bot that that ignore my robots.txt and
>>> proceeds to mirror my site, several times a day, burning up my
>>> upload bandwidth. They've moved it to 5 different addresses since
>>> midnight.
>>>
>>> I want to nail the door shut on the first attempted access by these
>>> AH's.
>>>
>>> Does anyone have a ready made script that can watch my httpd "other"
>>> log, and if a certain name is at the end of the line, grabs the ipv4
>>> src address as arg3 of the line, and applies it to iptables DROP
>>> rules?
>>>
>>> Or do I have to invent a new wheel for this?
>>>
>>> Basic rules that simplify it somewhat.
>>>
>>> 1. this is ipv4 only country and not likely to change in the future
>>> decade.
>>>
>>> 2. the list of offending bot names will probably never go beyond 50,
>>> if that many. 5 would be realistic.
>>>
>>> 3. the src address in the log is at a fixed offset, obtainable with
>>> the bash MID$ but the dns return will need some acrobatics involving
>>> the bash RIGHT$ function.
>>>
>>> 4. it should track the number of hits, and after so many in a /24
>>> block, autoswitch to a /16 block in order to keep the rules file
>>> from exploding.
>>
>> Is that not the same question you asked a while back, I then suggested
>> 'fail2ban' or using ip/nftables own capabilities?
>>
> Yes John. But explain to me what fail2ban is sopposed to do?
>
> Its running, but has failed to ban anything no matter what sort of 403's
> I return.
>
> Fail2ban has been running here for years, and in just sits there doing
> nothing, so if its as great a swiss army knife as others claim it to be,
> lets either make it work, or quit recommending it.
>
> I need something I can feed with a tee off the tail output, detect that
> it is one of the offending bots by name, and if so, apply its ipv4
> address to an iptables DROP rule.
>
>> It looks to me like you are making your life way much harder than it
>> should be.
>
> Fine, now show me how to make fail2ban do something usefull. I just
> rebooted because of a drive failure and found it couldn't be found
> running by htop, so I started it. Now make it do something usefull if
> its so great.
>

Assuming that I would be able to help you, I have no clue about your
setup nor how fail2ban is configured.

In general, when something does not work the way you want it to, it is
probably misconfigured.

If fail2ban or rate limiting in apache does not work for you have a look
at (1).

You could also use a frontend to iptables to implement hash limit module.


1)
https://poorlydocumented.com/2017/08/understanding-iptables-hashlimit-module/

--
John Doe

[toc] | [prev] | [next] | [standalone]

#229269

Hello,

On Thu, Dec 03, 2020 at 07:35:27AM -0500, Gene Heskett wrote:
> I've had it with a certain bot that that ignore my robots.txt and 
> proceeds to mirror my site, several times a day, burning up my upload 
> bandwidth. They've moved it to 5 different addresses since midnight.

This must be the third or fourth time we have been here with this
exact question from you. Every time the answers have been "Fail2Ban
and block by user agent". I don't know why you expect the answers to
change.

Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#229276

On Thursday 03 December 2020 21:13:35 Andy Smith wrote:

> Hello,
>
> On Thu, Dec 03, 2020 at 07:35:27AM -0500, Gene Heskett wrote:
> > I've had it with a certain bot that that ignore my robots.txt and
> > proceeds to mirror my site, several times a day, burning up my
> > upload bandwidth. They've moved it to 5 different addresses since
> > midnight.
>
> This must be the third or fourth time we have been here with this
> exact question from you. Every time the answers have been "Fail2Ban
> and block by user agent". I don't know why you expect the answers to
> change.
>
> Andy

Fail2ban does not come configured to do anything. In this case, not even 
waste cpu cycles. I've now read thru most of the configs, which may have 
been semi applicable in 2013, the date of its last update. But this, in 
case no one has noticed, is now the fading ragged edges of 2020.

Its .conf files do not mention, nor do they tell you how to edit them to 
work with the directory structures in common use today.  Someone who has 
kept their copy up to date with the ever changing web landscape may in 
fact have a working version, but my install isn't, its a waste of cpu 
cycles, not logging anything in the 20 some hours its been running 
except its nominally 2k startup stanza.

Tell me how to edit, AND enable the use of the apache-badbots.conf. isn't 
stated anyplace I've found so I've no clue how to add to the top line to 
include todays bad bots.  That's my 2nd bitch. Is that list basically a 
CSV but with a "|" as the comma?

Ditto, 1st bitch, what file do I edit to add 
todays /var/log/httpd/other_vhosts_access.log to its list of logs to 
watch.  That's the log file with the real data in it today.  And does it 
need enabled in another, different file.

Then, once its seeing the hits, how do I make it add to iptables/rules 
DROP list? Some "jail" file that I have to assume needs enabled. But the 
man pages do not explain that either. They might as well be written in 
swahili.

That would at least point it to the logs of interest today. And get some 
error messages that might guide the betterment of its configuration. 
THEN it might snowball into something usefull, but FIRST I need to make 
it read the right files to get it started.  Keeping fail2ban up to date 
is called "support", people, and I don't see any since 2013.

Thank you.  Stay safe and well everybody.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#229278

Hello,

On Fri, Dec 04, 2020 at 12:03:57AM -0500, Gene Heskett wrote:
> what file do I edit to add todays
> /var/log/httpd/other_vhosts_access.log to its list of logs to
> watch.  That's the log file with the real data in it today.  And
> does it need enabled in another, different file.

Again we have been down this avenue before, but I will try one last
time.

It seems quite likely that the bot you have a problem with has the
same user agent string, or a very small variation on the same
string. If so then you can block it just with Apache.

So, can you show us a few lines of logs from your
/var/log/httpd/other_vhosts_access.log of the accesses from the
offending bot(s)?

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#229279

On 12/4/2020 7:03 AM, Andy Smith wrote:
> Hello,
>
> On Fri, Dec 04, 2020 at 12:03:57AM -0500, Gene Heskett wrote:
>> what file do I edit to add todays
>> /var/log/httpd/other_vhosts_access.log to its list of logs to
>> watch.  That's the log file with the real data in it today.  And
>> does it need enabled in another, different file.
>
> Again we have been down this avenue before, but I will try one last
> time.
>
> It seems quite likely that the bot you have a problem with has the
> same user agent string, or a very small variation on the same
> string. If so then you can block it just with Apache.
>
> So, can you show us a few lines of logs from your
> /var/log/httpd/other_vhosts_access.log of the accesses from the
> offending bot(s)?
>

That is, increase your log verbosity (1) and and give us part of the
relevent lines.

Also if fail2ban is not working turn it off and other stuff that you
have put inplace to block those attacks, so we have something clean to
work with, apache is your first line of defence here.

1)  https://httpd.apache.org/docs/2.4/mod/core.html#loglevel

--
John Doe

[toc] | [prev] | [next] | [standalone]

#229281

On Friday 04 December 2020 01:34:52 john doe wrote:

> On 12/4/2020 7:03 AM, Andy Smith wrote:
> > Hello,
> >
> > On Fri, Dec 04, 2020 at 12:03:57AM -0500, Gene Heskett wrote:
> >> what file do I edit to add todays
> >> /var/log/httpd/other_vhosts_access.log to its list of logs to
> >> watch.  That's the log file with the real data in it today.  And
> >> does it need enabled in another, different file.
> >
> > Again we have been down this avenue before, but I will try one last
> > time.
> >
> > It seems quite likely that the bot you have a problem with has the
> > same user agent string, or a very small variation on the same
> > string. If so then you can block it just with Apache.
> >
> > So, can you show us a few lines of logs from your
> > /var/log/httpd/other_vhosts_access.log of the accesses from the
> > offending bot(s)?
>
> That is, increase your log verbosity (1) and and give us part of the
> relevent lines.
>
> Also if fail2ban is not working turn it off and other stuff that you
> have put inplace to block those attacks, so we have something clean to
> work with, apache is your first line of defence here.
>
> 1)  https://httpd.apache.org/docs/2.4/mod/core.html#loglevel
>
Page doesn't cover my stretch version. Paths are not the same for 
starters, yet I have 5 installs here from wheezy to buster to raspbian.  
All alike, in my looking around, but not THAT version.
> --
> John Doe


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#229280

On Friday 04 December 2020 01:03:34 Andy Smith wrote:

> Hello,
>
> On Fri, Dec 04, 2020 at 12:03:57AM -0500, Gene Heskett wrote:
> > what file do I edit to add todays
> > /var/log/httpd/other_vhosts_access.log to its list of logs to
> > watch.  That's the log file with the real data in it today.  And
> > does it need enabled in another, different file.
>
> Again we have been down this avenue before, but I will try one last
> time.
>
> It seems quite likely that the bot you have a problem with has the
> same user agent string, or a very small variation on the same
> string. If so then you can block it just with Apache.
>
> So, can you show us a few lines of logs from your
> /var/log/httpd/other_vhosts_access.log of the accesses from the
> offending bot(s)?
>
No.  Bad idea.

By publishing the name of the bot, it probably will be changed before the 
day is done. Far better IMO to tell me what files I need to edit, and 
the expected syntax of those edits. Much more universally usefull.  Or, 
update the package so it actually does something OOTB and let us run 
from there once we can actually see it working and or generating errors 
that will educate us, however poorly.
> Cheers,
> Andy


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#229282

On 12/4/2020 7:40 AM, Gene Heskett wrote:
> On Friday 04 December 2020 01:03:34 Andy Smith wrote:
>
>> Hello,
>>
>> On Fri, Dec 04, 2020 at 12:03:57AM -0500, Gene Heskett wrote:
>>> what file do I edit to add todays
>>> /var/log/httpd/other_vhosts_access.log to its list of logs to
>>> watch.  That's the log file with the real data in it today.  And
>>> does it need enabled in another, different file.
>>
>> Again we have been down this avenue before, but I will try one last
>> time.
>>
>> It seems quite likely that the bot you have a problem with has the
>> same user agent string, or a very small variation on the same
>> string. If so then you can block it just with Apache.
>>
>> So, can you show us a few lines of logs from your
>> /var/log/httpd/other_vhosts_access.log of the accesses from the
>> offending bot(s)?
>>
> No.  Bad idea.
>
> By publishing the name of the bot, it probably will be changed before the
> day is done. Far better IMO to tell me what files I need to edit, and

We did that to no avail.

I drop the ball on this one.

--
John Doe

[toc] | [prev] | [next] | [standalone]

#229283

On Fri, Dec 04, 2020 at 01:40:53AM -0500, Gene Heskett wrote:
> On Friday 04 December 2020 01:03:34 Andy Smith wrote:
> > Again we have been down this avenue before, but I will try one last
> > time.

[…]

> > So, can you show us a few lines of logs from your
> > /var/log/httpd/other_vhosts_access.log of the accesses from the
> > offending bot(s)?
> >
> No.  Bad idea.
> 
> By publishing the name of the bot, it probably will be changed before the 
> day is done.

It is beyond ridiculous to believe that a bot that is scanning the
entire Internet every day — and probably already published on
literally thousands of web sites that put up log analyses — is going
to find your single posting on debian-user and change its ways.

As predicted, you have zero interest in finding a practical solution
to your issue and so as promised, I am out.

I look forward to seeing your next try at this exact same thread.

Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#229291

On Fri, 4 Dec 2020, at 05:03, Gene Heskett wrote:

> Fail2ban does not come configured to do anything. In this case, not even 
> waste cpu cycles. I've now read thru most of the configs, which may have 
> been semi applicable in 2013, the date of its last update. But this, in 
> case no one has noticed, is now the fading ragged edges of 2020.

I was puzled when I read your plea yesterday, because 20 seconds with
Google found me

  https://www.fail2ban.org/wiki/index.php/Main_Page

Have you read all the info there?

Also there's a support maillist.  Why not join it and ask questions there?


-- 
Jeremy Nicoll - my opinions are my own.

[toc] | [prev] | [next] | [standalone]

#229304

On Friday 04 December 2020 05:47:40 Jeremy Nicoll wrote:

> On Fri, 4 Dec 2020, at 05:03, Gene Heskett wrote:
> > Fail2ban does not come configured to do anything. In this case, not
> > even waste cpu cycles. I've now read thru most of the configs, which
> > may have been semi applicable in 2013, the date of its last update.
> > But this, in case no one has noticed, is now the fading ragged edges
> > of 2020.
>
> I was puzled when I read your plea yesterday, because 20 seconds with
> Google found me
>
>   https://www.fail2ban.org/wiki/index.php/Main_Page
>
> Have you read all the info there?
>
> Also there's a support maillist.  Why not join it and ask questions
> there?

Thank you for that, mailing list subscribed.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#229294

[Multipart message — attachments visible in raw view] — view raw

On Vi, 04 dec 20, 00:03:57, Gene Heskett wrote:
> 
> Fail2ban does not come configured to do anything. In this case, not even 
> waste cpu cycles. I've now read thru most of the configs, which may have 
> been semi applicable in 2013, the date of its last update. But this, in 
> case no one has noticed, is now the fading ragged edges of 2020.

It appears updated regularly to me (not using it though).

$ rmadison fail2ban
fail2ban   | 0.8.13-1        | oldoldstable      | source, all
fail2ban   | 0.9.6-2         | oldstable         | source, all
fail2ban   | 0.10.2-2~bpo9+1 | stretch-backports | source, all
fail2ban   | 0.10.2-2.1      | stable            | source, all
fail2ban   | 0.11.2-1        | testing           | source, all
fail2ban   | 0.11.2-1        | unstable          | source, all


Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

#229285

On 2020-12-03 13:35, Gene Heskett wrote:
> I've had it with a certain bot that that ignore my robots.txt and
> proceeds to mirror my site, several times a day, burning up my upload
> bandwidth. They've moved it to 5 different addresses since midnight.
> 
> I want to nail the door shut on the first attempted access by these AH's.
> 
> Does anyone have a ready made script that can watch my httpd "other" log,
> and if a certain name is at the end of the line, grabs the ipv4 src
> address as arg3 of the line, and applies it to iptables DROP rules?
> 
> Or do I have to invent a new wheel for this?
> 
> Basic rules that simplify it somewhat.
> 
> 1. this is ipv4 only country and not likely to change in the future
> decade.
> 
> 2. the list of offending bot names will probably never go beyond 50, if
> that many. 5 would be realistic.
> 
> 3. the src address in the log is at a fixed offset, obtainable with the
> bash MID$ but the dns return will need some acrobatics involving the
> bash RIGHT$ function.
> 
> 4. it should track the number of hits, and after so many in a /24 block,
> autoswitch to a /16 block in order to keep the rules file from
> exploding.
> 
> Any help will be much appreciated. PM's in this case welcome as I can't
> see broadcasting our armament against these MF'ers being broadcast on a
> public list.
> 
> Thanks all.
> 
> Cheers, Gene Heskett

Let me offer you an alternative option. (Most) bots work by analysing 
the referrals on each page in your website. Right? So, why not add a 
link to a page that normal users will never visit (e.g. because they do 
not see the link and thus will never click on it), but will show up in a 
bot's analysis? That way you can monitor your logs for entries 
containing that page. Every entity requesting that specific URL is blocked.

HTH

HdV

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web