Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #195673 > unrolled thread

AI spam filter

Back to article view | Back to linux.debian.user

Contents

  AI spam filter Nicolas George <george@nsup.org> - 2018-05-13 13:10 +0200
    Re: AI spam filter Gregory Seidman <gsslist+debian@anthropohedron.net> - 2018-05-13 22:20 +0200
      Re: AI spam filter Nicolas George <george@nsup.org> - 2018-05-13 22:30 +0200
        Re: AI spam filter Curt <curty@free.fr> - 2018-05-14 11:50 +0200
          Re: AI spam filter Darac Marjal <mailinglist@darac.org.uk> - 2018-05-14 12:30 +0200
            Re: AI spam filter Curt <curty@free.fr> - 2018-05-14 14:10 +0200
              Re: AI spam filter Darac Marjal <mailinglist@darac.org.uk> - 2018-05-14 14:20 +0200
                Re: AI spam filter Abdullah Ramazanoğlu <ar018@yahoo.com> - 2018-05-15 00:10 +0200
                  Re: AI spam filter deloptes <deloptes@gmail.com> - 2018-05-15 00:20 +0200
            Re: AI spam filter rhkramer@gmail.com - 2018-05-14 18:20 +0200
              Re: AI spam filter deloptes <deloptes@gmail.com> - 2018-05-15 00:20 +0200
    Re: AI spam filter deloptes <deloptes@gmail.com> - 2018-05-15 00:40 +0200
      Re: AI spam filter conover@rahul.net (John Conover) - 2018-05-15 05:00 +0200

#195673 — AI spam filter

[Multipart message — attachments visible in raw view] — view raw

Hi.

For years I have been using bogofilter as my main spam filter. But
nowadays, neural networks and machine learning have made enormous
progress.

I wonder if there is somewhere in Debian a spam filter (working on the
same principles as bogofilter: a stand-alone program that reads a mail
from its input and outputs a diagnosis) based on these technologies and
that people find satisfactory.

Any advice?

Regards,

-- 
  Nicolas George

[toc] | [next] | [standalone]

#195689

Spamassassin has a Bayesian classifier (a standard AI technique) in
addition to its rules-base classification. You train it on positive (spam)
and negative (non-spam) messages. I've been using it for at least a decade
and I've been very happy with its increasing accuracy as I retrain it on
correct and incorrect classifications it's made. At this point I get few
errors either way.

--Greg

On Sun, May 13, 2018 at 12:03:01PM +0200, Nicolas George wrote:
> Hi.
> 
> For years I have been using bogofilter as my main spam filter. But
> nowadays, neural networks and machine learning have made enormous
> progress.
> 
> I wonder if there is somewhere in Debian a spam filter (working on the
> same principles as bogofilter: a stand-alone program that reads a mail
> from its input and outputs a diagnosis) based on these technologies and
> that people find satisfactory.
> 
> Any advice?
> 
> Regards,
> 
> -- 
>   Nicolas George

[toc] | [prev] | [next] | [standalone]

#195690

[Multipart message — attachments visible in raw view] — view raw

Gregory Seidman (2018-05-13):
> Spamassassin has a Bayesian classifier (a standard AI technique) in
> addition to its rules-base classification.

Thanks for the pointer. But as I explained, I already use bogofilter,
which is already a Bayesian filter (and IIRC much lighter than
Spamassassin). But Bayesian classification is a very limited form of AI.

What I am looking for is something that makes uses of the tremendous
theoretical and practical progress made in the last fifteen years.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#195703

On 2018-05-13, Nathaniel Suchy (Lunorian) <me@lunorian.is> wrote:
>
> Have any spam filters attempted to use neural networks? How well did
> that work out?
>

Apparently, they work quite well (but it seems you need mucho processing
power--an "enormous collection of computers"; "massive amounts of
hardware and massive amounts of data").

https://www.wired.com/2015/07/google-says-ai-catches-99-9-percent-gmail-spam/

https://www.wired.com/2015/04/jeff-dean/

Perhaps the absorption of a grain of salt might be indicated while
perusing these articles, I'm uncertain. Then again maybe a grain of salt
is always indicated no matter what the article or source.

[toc] | [prev] | [next] | [standalone]

#195708

[Multipart message — attachments visible in raw view] — view raw

On Mon, May 14, 2018 at 09:45:16AM +0000, Curt wrote:
>On 2018-05-13, Nathaniel Suchy (Lunorian) <me@lunorian.is> wrote:
>>
>> Have any spam filters attempted to use neural networks? How well did
>> that work out?
>>
>
>Apparently, they work quite well (but it seems you need mucho processing
>power--an "enormous collection of computers"; "massive amounts of
>hardware and massive amounts of data").
>
>https://www.wired.com/2015/07/google-says-ai-catches-99-9-percent-gmail-spam/
>
>https://www.wired.com/2015/04/jeff-dean/
>
>Perhaps the absorption of a grain of salt might be indicated while
>perusing these articles, I'm uncertain. Then again maybe a grain of salt
>is always indicated no matter what the article or source.
>

I'm no expert in AI, but it was my understanding that, while you need 
massive amounts of data and processing of that data to CREATE an AI 
model, the model itself can be relatively modest to run. This, I 
believe, is how smartphones can now boast "AI processing" for your 
photos and how Google's "auto-complete your emails" feature works.

So, in theory, someone - such as google - could produce a marketable 
version of their Great Email AI, which could run on single consumer PC 
and still benefit from the knowledge of what's spam and what's not.

Perhaps, though, where many Run-It-Yourself spam filters are lacking is 
in breadth of knowledge. Oftentimes, it is not JUST the text in the 
email that indicates a message is spam. Has it come from a domain which 
has only recently been created? Has the sender just sent the same, or 
nearly the same, message to umpteen different domains? Does the sender 
normally do this (e.g. it's a mailing list host) or has it just started 
(e.g. it's infected by a botnet). These days, spam filtering needs to be 
quite involved in the email server.

-- 
For more information, please reread.

[toc] | [prev] | [next] | [standalone]

#195712

On 2018-05-14, Darac Marjal <mailinglist@darac.org.uk> wrote:
>
> I'm no expert in AI, but it was my understanding that, while you need
> massive amounts of data and processing of that data to CREATE an AI
> model, the model itself can be relatively modest to run. This, I
> believe, is how smartphones can now boast "AI processing" for your
> photos and how Google's "auto-complete your emails" feature works.

But is the smartphone itself doing the actual processing (in apps
relying on *neural nets*), or is it only uploading the data to internet
servers that do the processing work and receiving the subsequent
results? 

[toc] | [prev] | [next] | [standalone]

#195714

[Multipart message — attachments visible in raw view] — view raw

On Mon, May 14, 2018 at 12:02:10PM +0000, Curt wrote:
>On 2018-05-14, Darac Marjal <mailinglist@darac.org.uk> wrote:
>>
>> I'm no expert in AI, but it was my understanding that, while you need
>> massive amounts of data and processing of that data to CREATE an AI
>> model, the model itself can be relatively modest to run. This, I
>> believe, is how smartphones can now boast "AI processing" for your
>> photos and how Google's "auto-complete your emails" feature works.
>
>But is the smartphone itself doing the actual processing (in apps
>relying on *neural nets*), or is it only uploading the data to internet
>servers that do the processing work and receiving the subsequent
>results?
>

The smartphone CAN do local AI processing. I find this video quite an 
accessible introduction to neural networks [MarI/O - Machine Learning 
for Video Games][https://www.youtube.com/watch?v=qv6UVOQ0F44&t=281s]. 
The video shows that, while it takes quite a bit of work to create the 
neural network (several plays through, categorising a run as successful 
or not, etc), the resulting logic can be quite simple ("If there is a 
block at co-ordinates X,Y, press the Up Key". "If there is a block at 
X,Y AND no block at W,Z, press the Left Key" etc).

Or, in other words, while it may take several CPU-decades to train an AI 
what a dog looks like, once it has learnt that, recognising a dog then 
becomes a trivial task.

-- 
For more information, please reread.

[toc] | [prev] | [next] | [standalone]

#195742

On Mon, 14 May 2018 13:16:01 +0100 Darac Marjal said:
 
> Or, in other words, while it may take several CPU-decades to train an AI 
> what a dog looks like, once it has learnt that, recognising a dog then 
> becomes a trivial task.

I can confirm to this. Once I was upto a pseudo-AI[0] project for stock market
portfolio optimization. It took months to optimize the optimizer (4 * AMD
Phenom 3400 MHz core at full load). Once it has been fine tuned, it was less
than a minute's job to calculate the next[1] buy & sell orders for a given
portfolio.

[0] "pseudo-AI" as it was a mixture of statistical computation and artificial
intelligence. [1] Hourly, daily, weekly, ... however one wants to use it.

Regards
-- 
Abdullah Ramazanoğlu

[toc] | [prev] | [next] | [standalone]

#195743

Abdullah Ramazanoğlu wrote:

> Once it has been fine tuned, it was less
> than a minute's job to calculate the next[1] buy & sell orders for a given
> portfolio.

Yes but this is more or less a closed domain, while for mailing things look
different.

[toc] | [prev] | [next] | [standalone]

#195733

On Monday, May 14, 2018 06:23:07 AM Darac Marjal wrote:
> I'm no expert in AI, but it was my understanding that, while you need
> massive amounts of data and processing of that data to CREATE an AI
> model, the model itself can be relatively modest to run. 

I believe the above.

> This, I
> believe, is how smartphones can now boast "AI processing" for your
> photos and how Google's "auto-complete your emails" feature works.

They might work that way (I have no knowledge of them), but I do know some 
other features on smartphones offload the work to a server.

AFAIK, google's voice recognition / transcription works that way (by sending 
the sound to a server and letting the server translate the voice to text), and 
then sending the text back to the smartphone.

[toc] | [prev] | [next] | [standalone]

#195744

rhkramer@gmail.com wrote:

> AFAIK, google's voice recognition / transcription works that way (by
> sending the sound to a server and letting the server translate the voice
> to text), and then sending the text back to the smartphone.

bingo!

and you accepted the license, which gives exclusive right to google, to
store and process the data for other purposes as well, so basically
everyone using it, is working for google for free and because of the amount
of data they acquire, their applications work better and better.

However as Microsofts AI bot showed, it is dangerous to leave such a thing
my it's own.

[toc] | [prev] | [next] | [standalone]

#195745

Nicolas George wrote:

> Hi.
> 
> For years I have been using bogofilter as my main spam filter. But
> nowadays, neural networks and machine learning have made enormous
> progress.
> 
> I wonder if there is somewhere in Debian a spam filter (working on the
> same principles as bogofilter: a stand-alone program that reads a mail
> from its input and outputs a diagnosis) based on these technologies and
> that people find satisfactory.
> 
> Any advice?
> 
> Regards,
> 

I doubt there is something meaningful beyond bayesian spam filter. It is the
most efficient way. Neural Network is a method, which would not apply very
well to spam detection as it is more sophisticated and leads to unjustified
overhead. This is my estimation only, but I spent few years studying some
aspects of AI in the domain of text and speech processing and I think I
would qualify for opinion.
If I recall correctly bayesian method is based on providing probability by
observing 1 item in context of 2 other items, which is scientifically
proven to be most efficient. I recall there were statistics that you gain
almost nothing if you add another 1 or 2 items to the context.
At that time neural networks were introduced and we had few exercises, so I
think I also know how basically NN works. 

Regarding spam I think sooner or later more mail providers will introduce
more restrictive DKIM rules and spam should be reduced. So if you want a
modern method of fighting spam install the dkim plugin to exim and try it
out. It may take some effort to configure ... but it seems you are looking
for a project to keep yourself busy. Condition is however to have a valid
mail domain - it won't work with localhost or your ISPs IP.

regards

[toc] | [prev] | [next] | [standalone]

#195756

deloptes writes:
> Nicolas George wrote:
> 
> I doubt there is something meaningful beyond bayesian spam filter. It is the
> most efficient way. Neural Network is a method, which would not apply very
> well to spam detection as it is more sophisticated and leads to unjustified
> overhead. This is my estimation only, but I spent few years studying some
> aspects of AI in the domain of text and speech processing and I think I
> would qualify for opinion.
> If I recall correctly bayesian method is based on providing probability by
> observing 1 item in context of 2 other items, which is scientifically
> proven to be most efficient. I recall there were statistics that you gain
> almost nothing if you add another 1 or 2 items to the context.
> At that time neural networks were introduced and we had few exercises, so I
> think I also know how basically NN works. 
>

Its very old, (c. 1994, when "bugless" added scoring to recipes,) but
procmail(1) can be used for Bayesian UCE filtering, (or very close
approximation thereof.) See:

    http://www.johncon.com/john/StochasticUCEDetection/

In addition, it can analyze message's metadata, (i.e., Received:
records, etc.,) and, importantly, handle whitelists, etc.

    John

-- 

John Conover, conover@rahul.net, http://www.johncon.com/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web