MERO

How would you rate MERO?

Excellent...a Must Have!
8
42%
Helpful, useful, worth it.
9
47%
I could take it or leave it.
0
No votes
I tried it, but didn't like it.
1
5%
It's useless...don't bother.
1
5%
 
Total votes: 19
Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

MERO

Postby Saskatchewan » Fri Dec 23, 2005 7:50 am

Full Name: "MailEnable Repeat Offenders"
I call it MERO.

Download: free of charge for both commercial and personal use.


What is it?

MERO helps to prevent false positives in MailEnable's SMTP-DENY access list (sometimes called the "auto-ban" feature). MERO maintains a database of repeat offenders so you can be assured that the least offensive IP addresses are released back into circulation.

This utility should be run as a scheduled event (a couple of times per day perhaps) and it will help maintain the list of IP addresses which are denied access to your SMTP server when you have selected MailEnable's option to "Add to denied IP Addresses if number of failed commands or recipients reaches X ".

MailEnable will automatically deny access for a single "offense" and this sometimes causes valid computers (i.e. Hotmail.com mail servers for example) to be inadvertantly denied access to your SMTP server -- forever! MERO resolves this problem.


Compatibility?

Windows 2000 | 2003 | XP (I presume but I've tested and used MERO only with Windows XP and Windows Server 2003.)
MailEnable - All Editions (I presume but I've tested and used MERO only with Standard Edition.)


Installation?

Requires only a few minutes. It's all explained in the ReadMe document in the downloadable zip package.


Why MERO?

Because: It's been suggested a number of times in these forums and in MailEnable's own documentation that server administrators should "empty" or "selectively remove" IP addresses from the SMTP-DENY list. That always struck me as a half-baked idea and it opens a lot of questions: How should I know which addresses to release? and which addresses cause the biggest problem?

Because: I want to know that my server is protected from the IP addresses that are running dictionary attacks against my SMTP service...but that my server allows connections to legitimate mail servers. MERO makes this easy.


Other info?

It's all explained in the ReadMe document -- or if I've missed anything then please raise questions here in this forum or contact me by email.


Cheers.

labsy
Posts: 148
Joined: Sun Nov 16, 2003 6:49 am
Location: Slovenia
Contact:

Postby labsy » Mon Jan 30, 2006 6:47 am

Excellent work!
The only thing I miss, is kinda WHITELIST - list of IP addresses, which never get banned. ME's whitelist does not work and I always find my own secondary mail server on deny list.

Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

Postby Saskatchewan » Sat Feb 11, 2006 9:10 pm

If MailEnable's whitelist features aren't working properly then that issue would best be addressed at the source (by the MailEnable developers) -- I'd prefer to keep MERO out of that loop and focus its features on the SMPT-DENY list.

I haven't experienced any problems with MailEnable's whitelist...but I also haven't spent much energy studying it.

In any event, thank your for you interest and encouragement! I'm glad that you've found MERO helpful.

Regards.

Colin
Posts: 17
Joined: Tue Oct 28, 2003 4:56 pm
Location: Gillingham, Kent. UK

Postby Colin » Thu Jun 22, 2006 8:57 pm

Hi I installed mero a couple of days ago, and I wondered how long should it take to start paroling IP's?
I have been watching the logs and it goes through all of the processes, but no IP's have been released even though their threat level is 1. I even incressed the schedule to hourly to see if that would speed things up.

Thanks for any info

Colin

Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

Postby Saskatchewan » Fri Jun 23, 2006 12:46 am

Colin wrote:Hi I installed mero a couple of days ago, and I wondered how long should it take to start paroling IP's?


Hello Colin,

Running MERO every hour is likely too frequent. I run MERO three or four times per day and I find that it's sufficient. MERO begins working immediately...but it's most effective after a few weeks (and more effective as time goes by).

MERO should parole a percentage of IPs right away and the log files should tell you how many are 'on parole'. The log file contains a great deal of information, but the most helpful information is in a section which looks like this (here's an example log from my mail server):

EXAMPLE FROM LOG FILE:
18 records were added.
1004 records were updated.
MERO found -1 'repeat offenders' and their ThreatLevel will be increased.

I will scan the MERO database for stale records.
Some older records may be released to learn if they re-offend.
And some IP addresses that have proven NOT to be 'Repeat Offenders' will be released and removed entirely from MERO's database.

0 records have been removed/redeemed forever by MERO.

304 records have been allowed an opportunity to redeem themselves. Note that this does not mean they are immediately allowed access - it only means that MERO will place them in a queue and may be released at some point in the near future. If they re-offend, then their ThreatLevel will increase and may be considered incorrigible and banned forever.

Dropping temporary table...Done.

I will prepare a new list of banned IP addresses with the new data.
The list will include:
- IP addresses which are considered 'incorrigible'.
- IP addresses which are considered a current threat.

The list will exclude:
- IP addresses which are not considered a threat.
- and 131 IP addresses which will be released 'on parole'. (These addresses will be allowed access to the SMTP server and MERO will watch to see if they re-offend. If they do, their TheatLevel will increase and MERO may decide that they are incorrigible.)

Of 15312 total records managed by MERO, 1195 will be returned to Mail Enable's SMTP-DENY list.
14117 addresses are either 'on parole' or are not considered a current threat.


From that log file you can see that 131 IP addresses will be released 'on parole'. That line indicates that, in this batch, 131 addresses will be added to the others that are already on parole. Note also that 14117 addresses in total have been released on parole since I installed MERO on my server and that of the fifteen thousand IP addresses that have been blocked, only 1195 have re-offended.

Perhaps you could copy-n-paste a section in your own log file into this forum so I can see? I'm most interested in seeing the section in your log file that looks like my example above.

Regards.

Colin
Posts: 17
Joined: Tue Oct 28, 2003 4:56 pm
Location: Gillingham, Kent. UK

Postby Colin » Fri Jun 23, 2006 12:05 pm

Hi Saskatchewan, thanks for getting back to me so quickly.
I have copied a bit of a log below which seems to show my problem

It appears to me (although I could be wrong so I bow to your superior knowledge) that

one new address was added to list making 6 in total
one was going to be released by mero on parole.
Yet all six ip's were returned to the smtp denied list :?:




1 records were added.
5 records were updated.
MERO found 0 'repeat offenders' and their ThreatLevel will be increased.

I will scan the MERO database for stale records.
Some older records may be released to learn if they re-offend.
And some IP addresses that have proven NOT to be 'Repeat Offenders' will be released and removed entirely from MERO's database.

0 records have been removed/redeemed forever by MERO.

0 records have been allowed an opportunity to redeem themselves. Note that this does not mean they are immediately allowed access - it only means that MERO will place them in a queue and may be released at some point in the near future. If they re-offend, then their ThreatLevel will increase and may be considered incorrigible and banned forever.

Dropping temporary table...Done.

I will prepare a new list of banned IP addresses with the new data.
The list will include:
- IP addresses which are considered 'incorrigible'.
- IP addresses which are considered a current threat.

The list will exclude:
- IP addresses which are not considered a threat.
- and 1 IP addresses which will be released 'on parole'. (These addresses will be allowed access to the SMTP server and MERO will watch to see if they re-offend. If they do, their TheatLevel will increase and MERO may decide that they are incorrigible.)

Of 6 total records managed by MERO, 6 will be returned to Mail Enable's SMTP-DENY list.0 addresses are either 'on parole' or are not considered a current threat.


Am I reading this correctly?

Yes I realise that I am running mero too frequently, but I was just becoming worried that no IP's were being released so I was just tring to hurry the process up to reasure myself that it was working.

Many thanks for your thoughts
Colin

Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

Postby Saskatchewan » Fri Jun 23, 2006 3:51 pm

Hello Colin,

I've hunted in the code for the source of the problem. I can tell you these things:

1. MERO is working as designed. It's working properly.

2. MERO is designed to release 10% (or a maximum of 100) of the IP addresses on parole at one time. The problem you're seeing is simply that 10% of SIX is less-than-one. That brings me to the next point...

3. If you have only six IP addresses on the SMTP-DENY list then either you're operating a mail server in a VERY controlled environment or your 'auto-ban' settings are much too liberal. More on that later...

4. I have discovered a slight error in the code however...
and 1 IP addresses which will be released 'on parole'

The code which prints THAT message to the log was wrong. That number ("1" in your case) is produced by a simple equation but I had mixed up or misunderstood the way that Visual Basic rounds numbers. That number *should have been* be zero. This little error wouldn't have adverse effects on MERO's operation -- but it has caused the output in the log file to be rather misleading.

_________________________________

So...here are two concluding thoughts:

First: I recommend you have a look at your SMTP/Security properties. In the box called "Drop a connection when the failed number of commands or recipients reaches: X" you might want to enter a smaller number there. Why? Because if your SMTP server has been operating for a while then it's surprising to me that only six IP addresses have been blocked by Mail Enable. You could make that settings more restrictive -- particularly now that you've got MERO involved to help manage the SMTP-DENY list.

Second: Each time MERO runs, it will release 10% of the blocked IP addresses on parole. I think my logic is sound...and this policy is effective except where there are fewer than 10 IP addresses. As you're seeing, MERO cannot release a fraction of one IP address -- I'll be putting some more thought into this to see if there's a logical way to address this shortcoming.

In any event, as a result of your inquiry and my investigation I have addressed that slight error (described above in point #4) and updated the code slightly. You can get the updated file by downloading MERO from my site again. Here. The only files that have changed in this update is "MERO.vbs" and the ReadMe file -- you can simply extract MERO.vbs from the zip and overwrite the current copy that you're using.

Regards.

Colin
Posts: 17
Joined: Tue Oct 28, 2003 4:56 pm
Location: Gillingham, Kent. UK

Postby Colin » Mon Jun 26, 2006 10:23 am

:shock: wow what a detailed reply! Thanks Saskatchewan 8)
That all makes perfect sense, I knew there would be a good reason, and I do not think it is too much of a shortfall in your code just the way I was interpriting the documentation.
Anyway thanks again for that, now I understand whats going on.

My SMTP deny list is small because I have manualy cleared it out regularly in the past, but now I am using another bit of code that someone (Funmeister) on the MEFilter forum has put together which strips IP out of e-mail headers and puts them in the deny list based on criteria you can set up within a mailenable filter. We are testing it out with a honey-pot email address planted on our websites just to attract spam, and therefore ban the IP's, so as you can see your "Mero" makes the perfect partner to this to release one off' offenders.

Colin

Smurf
Posts: 516
Joined: Thu Apr 22, 2004 6:42 pm

Postby Smurf » Mon Jun 26, 2006 7:14 pm

Firstly. Great bit of code Saskatchewan.

We've been using your script for over 5 months. We've recently spotted that the script is eating up quite a lot of CPU time whilst it crunches away at the data. Due to the size of the SMTP deny list and Dbase it now takes roughly 7-8 minutes to complete and during this time it appears to take 100% CPU.

Is there anyway of running the VB script on a low priority?

Closing Mail Enable SMTP-DENY configuration.
==================================================
MERO is finished: 26/06/2006 09:07:03
Required 421.9219 CPU seconds.
==================================================

Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

Postby Saskatchewan » Wed Jun 28, 2006 4:24 pm

Hello Smurf,

Smurf wrote:Due to the size of the SMTP deny list and Dbase it now takes roughly 7-8 minutes to complete and during this time it appears to take 100% CPU.


422 seconds is an extremely long time for the process to operate. I suspect that your database is filled with many thousands of IP addresses. On my server, MERO requires about 90 seconds to operate and MERO is currently managing about 16000 addresses.

So, I have two suggestions:

1. I highly recommend that you should incorporate a blacklist like Spamhaus (if you haven't already). I've done this on my server and it has cut-down the number of bogus connections dramatically. So...incorrigible SMTP servers are blocked before they can barrage Mail Enable with dictionary attacks. This will help to keep your SMTP-DENY list to a minimum.

2. The process which usually takes the most CPU (and time) is not MERO's actual operation, but creating the log file. In other words, if you were to run MERO without creating the log file, then you'd be able to eliminate all steps to create the log file then repeatedly write each line and each IP address to the log file. I suspect that you'd immediately see a huge improvement in performance and speed of MERO's process.

MERO can be executed from a command line 'with' or 'without' logging. Here's an example command line which simply executes MERO:

C:\> cd C:\Program Files\Mail Enable\MERO\
c:\Program Files\Mail Enable\MERO> MERO.vbs

The 'logging' option can be turned on or off by the second argument. Like this:

C:\> cd C:\Program Files\Mail Enable\MERO\
c:\Program Files\Mail Enable\MERO> MERO.vbs 1,0

The ZERO in the above command line indicates that logging should be disabled. The ReadMe file contains more information about the command-line arguments.

Perhaps MERO should operate without logging on your server most of the time...then you could execute MERO once per week with logging enabled just so you'd be able to keep an eye on things.

Smurf wrote:Is there anyway of running the VB script on a low priority?


Not that I am aware of. I didn't build any option within the code to alter the process's priority -- and (although I've search high and low) I do not believe that a .VBS file can be executed with low priority from the command line. .VBS files can operate within the CSCRIPT interface as well...but I haven't found any evidence that a the CSCRIPT interface can be run in low priority from a command line either. Sorry.

The good news is this: .VBS scripts should run in 'Normal' priority by default on most systems. That means that they will share the CPU with other process that are currently running -- if the CPU can give them 100% then they'll take 100% but if one or more other processes requires the CPU also then all the processes will share equally.

I hope this helps.

Colin
Posts: 17
Joined: Tue Oct 28, 2003 4:56 pm
Location: Gillingham, Kent. UK

Postby Colin » Thu Jun 29, 2006 7:46 am

Hi Saskatchewan,
I'm back again, sorry to be a pain, but after running mero for a few more days with your newly posted code it is still not putting any IP's on parole.
Am I doing something wrong? I have posted the latest log entry below.
Colin


0 records were added.
25 records were updated.
MERO found 0 'repeat offenders' and their ThreatLevel will be increased.

I will scan the MERO database for stale records.
Some older records may be released to learn if they re-offend.
And some IP addresses that have proven NOT to be 'Repeat Offenders' will be released and removed entirely from MERO's database.

0 records have been removed/redeemed forever by MERO.

0 records have been allowed an opportunity to redeem themselves. Note that this does not mean they are immediately allowed access - it only means that MERO will place them in a queue and may be released at some point in the near future. If they re-offend, then their ThreatLevel will increase and may be considered incorrigible and banned forever.

Dropping temporary table...Done.

I will prepare a new list of banned IP addresses with the new data.
The list will include:
- IP addresses which are considered 'incorrigible'.
- IP addresses which are considered a current threat.

The list will exclude:
- IP addresses which are not considered a threat.
- and 3 IP addresses which will be released 'on parole'. (These addresses will be allowed access to the SMTP server and MERO will watch to see if they re-offend. If they do, their TheatLevel will increase and MERO may decide that they are incorrigible.)

Of 25 total records managed by MERO, 25 will be returned to Mail Enable's SMTP-DENY list.
0 addresses are either 'on parole' or are not considered a current threat.

Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

Postby Saskatchewan » Fri Jun 30, 2006 9:14 pm

Hello all,

As a result of communication with Colin (from previous post) I have identified a minor problem which effects MERO users who have less than 100 IP addresses in their MERO database.

If MERO in your system has more than 100 IP addresses then this problem does not effect you. If you have less than 100 IP addresses in MERO's database, then I recommend you download the most recent copy of the MERO.vbsfile.

Regards.

Colin
Posts: 17
Joined: Tue Oct 28, 2003 4:56 pm
Location: Gillingham, Kent. UK

Postby Colin » Tue Jul 04, 2006 8:40 pm

Hi All,

Thanks to Saskatchewan for all his efforts I can confirm that mero is now working 100% for me too.
I think maybe I have got a lucky mail server because even after 5 years we only had a very low number of denied IP's, but I wanted use another bit of code to block spammers by extracting their IP's from the mail headers sent to addresses set up specifically for this and also to ban users of certain words. Because this is a small company mailserver I felt that managing the denied list was importent so I wanted to have mero working for me first.

Anyway sorry to cause so much trouble, thanks again and I hope that maybe someone else will also benefit from the revision that I prompted.

This software and author certainly get my vote!

Regards
Colin
8)

andyhowie
Posts: 13
Joined: Wed Feb 22, 2006 3:04 pm

MERO Stopped working

Postby andyhowie » Thu Nov 16, 2006 7:20 pm

Is anyone else having trouble with MERO?

I noticed that about a month and a half ago it stopped running correctly, but did not notice the error until the other day.

Here is the log:
********
==================================================
MERO Starting: 11/16/2006 2:06:59 PM.
I will report to command window.
I will log my activity.
I will use the database at D:\MailEnable\MERO\MERO.mdb.
I will release a maximum of 100 addresses (or 10%, whichever is fewer) to learn if they re-offend.
Pausing for 10 seconds to display this message...




Opening MERO database...Done.
I will retreive IP addresses that are currently banned by Mail Enable.
Creating temporary recordset...Done.
Begin reading records.
Opening Mail Enable SMTP-DENY configuration...Done.
********

It does not appear to be picking anything up?

Any suggestions would be wonderful.

Saskatchewan
Posts: 27
Joined: Fri Sep 09, 2005 5:54 am
Location: Saskatchewan

Postby Saskatchewan » Fri Nov 17, 2006 5:32 am

Hello andyhowie,

If you open the MailEnable Administrator and look in:
- Servers > Your server > Connectors > SMTP > "Properties"
- then the "Inbound" tab > click the "Access Control..." button
- are there any IP addresses listed in the "Access Control" dialogue box?

The problem you're having might be explained if there are no IP addresses listed there (which is the SMTP-DENY list).

If that doesn't provide any clues, then if you have Microsoft Access on your computer then I'd be interested to know more about the records in the "meroSMTPDENY" table...are there any records? If so, perhaps you could cut-n-paste a sampling of those to help me understand the problem you're having better.

Who is online

Users browsing this forum: No registered users and 5 guests