new spamassassin script - Page 3

**pseudoswede** · July 31st, 2007, 6:18 PM

Another thing...

How can I have this script automagically remove spam that is older than, say, 30 days?

Thanks in advance.

**jdcopelin** · August 1st, 2007, 5:26 PM

Hello,

Thanks for trying out my script. I will try to answer your questions.

Quote:

Originally Posted by pseudoswede

Okay, I'm testing this out on one of my domains. I ran the cron job just now, and I got this result... Is this running correctly?

Unfortunately, no. There are some important lines missing from the output (see one of my previous post in this thread for an example of intended output).

Quote:

Here are my user-configurable variables...

Code:

# 1. spammassassin spam delivery folder name
MYSPAM=SPAM

# 2. learning spam (spam that arrived in the inbox, missed by spamassassin
MYLEARNSPAM=SPAM

# 3. ham (spam which got marked as spam when it was a genuine email
MYHAM=HAM

You need to put a "." at the start of the name of each folder. This is just the way maildir stores the actual folder and the script has to search for the actual name of the folder.

Quote:

How can I have this script automagically remove spam that is older than, say, 30 days?

The script sort of already does this. All messages present in a (spam) folder since the last login get moved to the specified archive folder SPAMDIR=$HOME/spamdb HAMDIR=$HOME/hamdb after 7 days. Somewhere around line 85 is this line of code:

Code:

SPAM_FILES=`find $SPAMFOLDER/cur -name \*, -mtime +7 -type f`

Just modify the +7 to +30 to change it to 30 days.
If you don't want to archive the files, then change the line of code a few lines down from 85

Code:

mv -f --target-directory=$SPAMDIR $SPAM_FILES

to

Code:

rm $SPAM_FILES

Feel free to ask if you have any more questions.

Cheers
Jonathan

**pseudoswede** · August 2nd, 2007, 5:38 PM

Thanks for the help.

It kind of worked, but I do have two error messages...

Code:

Running jicoweb.com spam assassin training script

JicoScript learnspam script  0.6.20070609
SpamAssassin version 3.2.2

Currently running as user: ****

Checking spam and ham repositories exist...
/home/****/spamdb exists!
/home/****/hamdb exists!

Learning from SPAM

        /home/****/mail/****/mailtrap/.SPAM
                learning from spam ...
                Learned tokens from 0 message(s) (2 message(s) examined)
                learning from old spam ...
                /home/****/script/learnspam: /usr/bin/sa-learn: /usr/bin/perl: bad
interpreter: Argument list too long
/home/****/script/learnspam: line 94: /bin/mv: Argument list too long

        /home/****/mail/****/innebandy/.SPAM
                learning from spam ...
                Learned tokens from 351 message(s) (365 message(s) examined)
                learning from old spam ...
                Learned tokens from 14 message(s) (14 message(s) examined)

Learning from uncaught SPAM

        /home/****/mail/****/mailtrap/.SPAM
                learning from messages ...
                /home/****/script/learnspam: /usr/bin/sa-learn: /usr/bin/perl: bad
interpreter: Argument list too long
/home/****/script/learnspam: line 119: /bin/mv: Argument list too long

        /home/****/mail/****/innebandy/.SPAM
                learning from messages ...
                Learned tokens from 1 message(s) (1 message(s) examined)

Learning HAM
 
Synchronising spam database
bayes: synced databases from journal in 0 seconds: 154 unique entries (154 total
entries)
expired old bayes database entries in 19 seconds
157777 entries kept, 0 deleted
token frequency: 1-occurrence tokens: 71.65%
token frequency: less than 8 occurrences: 18.03%

Done

(username and domain name have been removed)

There are probably around 1000 spams that are older than 30 days in that folder.

The line(s) in question are..

Code:

mv -f --target-directory=$SPAMDIR $SPAM_FILES

**jdcopelin** · August 3rd, 2007, 6:45 PM

Quote:

Originally Posted by pseudoswede

Thanks for the help.

It kind of worked, but I do have two error messages...

Code:

learning from old spam ...
/home/****/script/learnspam: /usr/bin/sa-learn: /usr/bin/perl: bad
interpreter: Argument list too long
/home/****/script/learnspam: line 94: /bin/mv: Argument list too long

There are probably around 1000 spams that are older than 30 days in that folder.

The line(s) in question are..

Code:

mv -f --target-directory=$SPAMDIR $SPAM_FILES

Oh no! That looks like a bug in my script (or limitation of how many files you can pass as a parameter in the command shell). Unfortunately my home PC has just suffered a hard drive failure so I don't have access to any of my files right now to investigate this and fix it. As soon as it is fixed, I will look into this for you.

Cheers
Jonathan

**pseudoswede** · August 3rd, 2007, 6:56 PM

I'm a software tester by trade, so I like to break things. Whether on purpose or accidentally.

Unfortunately, I'm not a developer, so I can't fix them.

Thanks for your help, I appreciate it.

**jdcopelin** · August 14th, 2007, 5:29 PM

Quote:

Originally Posted by pseudoswede

I'm a software tester by trade, so I like to break things. Whether on purpose or accidentally.

Unfortunately, I'm not a developer, so I can't fix them.

Thanks for your help, I appreciate it.

pseudoswede, I haven't forgotten about you! My PC has only just come back online after my HDD crash - my DVD drive failed as well so couldn't even restore a backup until I got a new drive fitted

Anyway, I have done some more testing and the only way around the command line arguments limit that I can find will make the whole script grossly inefficient. If you know BASH scripting I think the only way would be to call sa-learn and mv individually for each maildir message file that the "find" command lists? That will definitely slow the script down.

I think I can reduce the files sent to sa-learn by using grep to filter out spam already learnt as spam (assuming false negatives and caught spam are kept in the same folder).

Any comments are welcome!

Cheers
Jonathan

**Bigjohn** · August 14th, 2007, 6:01 PM

Does the Spam Assassin WIKI help this process at all?
http://wiki.apache.org/spamassassin/BayesInSpamAssassin

We've gone from a 10 line simple, efficient script on MBOX to a convoluted mess of code due to maildir...*sigh*

**jdcopelin** · August 15th, 2007, 9:10 AM

Quote:

Originally Posted by Bigjohn

We've gone from a 10 line simple, efficient script on MBOX to a convoluted mess of code due to maildir...*sigh*

Sorry you think my code is a "convoluted mess". My post in the thread that you and Cowboy contributed heavily to was a fix to get the script to run on a server that switched from MBOX to MAILDIR. The script here does a lot more than that original script. Unfortunately it has fallen foul of a kernel limitation in the size in memory arguments passed to programs via the command line.

I am about to go on holiday so won't be reading the forum for about a fortnight, but when I get back I will look through older scripts I based on Cowboys MBOX version and post it here.

Cheers
Jonathan

**Bigjohn** · August 15th, 2007, 10:49 AM

Quote:

Originally Posted by jdcopelin

Sorry you think my code is a "convoluted mess". My post in the thread that you and Cowboy contributed heavily to was a fix to get the script to run on a server that switched from MBOX to MAILDIR. The script here does a lot more than that original script. Unfortunately it has fallen foul of a kernel limitation in the size in memory arguments passed to programs via the command line.

I am about to go on holiday so won't be reading the forum for about a fortnight, but when I get back I will look through older scripts I based on Cowboys MBOX version and post it here.

Cheers
Jonathan

Jonathan - convoluted mess was not at all ment to be a disparagement, I hope you'll understand. My point was to simply state that something very elegant and simple as the original script cannot work due to the implementation of MAILDIR, enforcing further my conclusion that the actual "benefits" of maildir are probably limited to people who want GMAIL sized mail boxes and folders on their servers, where the limitations (speed of search, spam-assassin training, etc.) are foisted off on all.

Enjoy your holiday!

July 31st, 2007, 6:18 PM	#19 (permalink)
pseudoswede Surpass Fan Comfy Contributor Joined in Jun 2003 Lives in Denver Hosted on D9 142 posts Gave thanks: 4 Thanked 3 times	Another thing... How can I have this script automagically remove spam that is older than, say, 30 days? Thanks in advance. __________________ "In the end, everything will be fine - if it is not fine, it is not the end." PseudoSwede larvez.com Dime9

August 3rd, 2007, 6:56 PM	#23 (permalink)
pseudoswede Surpass Fan Comfy Contributor Joined in Jun 2003 Lives in Denver Hosted on D9 142 posts Gave thanks: 4 Thanked 3 times	I'm a software tester by trade, so I like to break things. Whether on purpose or accidentally. Unfortunately, I'm not a developer, so I can't fix them. Thanks for your help, I appreciate it. __________________ "In the end, everything will be fine - if it is not fine, it is not the end." PseudoSwede larvez.com Dime9 Last edited by pseudoswede; August 3rd, 2007 at 6:57 PM..

August 14th, 2007, 6:01 PM	#25 (permalink)
Bigjohn minor deity Super #1 Joined in Apr 2004 Lives in Georgia Hosted on XEON 7,395 posts Gave thanks: 28 Thanked 94 times	Does the Spam Assassin WIKI help this process at all? http://wiki.apache.org/spamassassin/BayesInSpamAssassin We've gone from a 10 line simple, efficient script on MBOX to a convoluted mess of code due to maildir...sigh __________________ Proud to be a Surmunity Mod! XEON PASS60 PASS61 Make a fundamental difference! My Sites: Curious about Brewing Beer? Join the community! >>>>> Some Change is GOOD! Keep your paycheck! Support the Fair Tax Get into an Art museum Victorian London It's your brain -*ON WEB* - mybrainhost.com (under development) What SHOULD Government do? Much Less than it Does!

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Rate This Thread
Rate This Thread:

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)