I ‘m already using a good email spam filter: SpamBayes, so I decided to test bayesian filtering for the spam on this blog too.

I decided to give Reverend a try:

from reverend.thomas import Bayes

SPAM_DB='spam.bayes'
guesser = Bayes()

# load the spam DB
try:
    guesser.load(SPAM_DB)
except IOError:
    print "Creating a new spam filter database"
    guesser.save(SPAM_DB)

def train_spam(text):
    guesser.train('spam',text)
    guesser.save(SPAM_DB)

def train_ham(text):
    guesser.train('ham',text)
    guesser.save(SPAM_DB)

# try to guess the spam / ham ratio of a text
def guess(text):
    spam = 0
    ham = 0
    value = guesser.guess(text)
    for o in value:
        if o[0] == 'ham': ham = o[1]
        if o[0] == 'spam': spam = o[1]
    return (ham,spam)

Small, and really simple module no ? The next step, simply add a ‘spam’ and ‘ham’ attributes on your comment post. And add two methods to train the comment as a spam or a ham.. And of course, only display comments which have a good ratio ( >1) ham/spam. This took me about 1 hour to implement…

After a week, of train, this is working very fine, not a single false positive, and it filter every spam since the first trains. As I get around 20 spams post per day, this is quite a good news ;)

Enjoy Bayesian ?

Little example of link:

Update: Hum a little variant now, I get the same w/ bloglines ..

http://www.kaotonik.net/zope/thk/portal_memberdata/portraits/mv#viagra

http://www.bloglines.com/blog/harrison

This is a kind of free hosting ;) So the next stuff is to post a link on blogs to this page. Now why are he doing this kind of hack ? I guess this avoid to be blacklisted by some antispam rules on certain blog software and allow to play with the Google Rank …

Now, i need to setup a mailing list manager… and issues are back again :( I read the mailman howto, and no, it doesn’t support virtual-hosting out-of-the-box. Play with a bunch of alias? no thanks… I can use another one of course, but intregation w/ postfix is not so-trivial for most of them.

So why not write a full featured SMTP daemon in Py? We will get:

• a system that can scale easily ..
• support for most of db-backend without a pain
• can embed spambayes .. or any other filtering system in only one piece of code ..
• only one users configuration (not to use LDAP or others to glue..)

Right now, if you want to have a good mail system you need to use a bunch of stuffs, written in a bunch of langage .. and use a bunch of tricks to glue each others… But everybody wants a nice/simple mail system ?

So the first try:

• install spamassassin :) .. This mail filtering is just a bulshit ! Even w/ the training done on a mail, it achieve to deliver it as ‘unsure spam’ !!
• put spamassin away .. but keep piece :)

I first decided to write another client / server for spambayes. but looking at all the stuff writen in spamc (spamassassin client) i discover i will need a lot of nights (i’m not a C guru . even if the little try i wrote works perferctly )

Nice try, but why i shouldn’t simply write a server that use spamc as client ? .. could be really easy and efficient too (spamc is really efficient) ..

i just finish to write this and submit to the dev list