SOLVED - please help, server hacked (postfix, mailq is filling up FAST)

I just sat down to find my server was buggered!

One of my sites, ourshore.ca is sending spam!!! I took the site off line (it is Joomla) and just stopped Postfix and Dovecot.

The Postfix Mail Queue is filling up with 100’s of emails every few minutes.

How do I find the problem and fix it?

Howdy,

Now that you’ve disabled the site, are you continuing to get more spam in the mail queue?

If so, that may mean you’re getting spam via a source other than just the site… that may mean one of the email accounts was broken into.

However, you should be able to determine that from the headers of the spam messages what the actual source is.

Once you’ve reviewed the headers, you may want to clear out the Postfix queue.

-Eric

The site is back up as it made no difference.

This came from a brute force attack, that is very apparent in the mail.log

My Server Provider just suggested to lock down with the firewall. Not sure how to do that.

If you found out which account/site is/was being used to send the spam (authentication headers and mail log entries should provide that info), it should suffice to lock down the respective account and then clear out the mail queue.

I wouldn’t really know what “locking down with the firewall” is supposed to mean or help…

the user it is using is ourshore.ca which is listed under users with login access of Email, FTP, and SSH

Currently the onlything ourshore.ca is doing is accessing a database.

How do I remove the EMAIL access? (this is the default user created by virtualmin when the virtual server was created)

EDIT: I have disabled the entire virtual server, and this has stopped the SPAM, but of course the website and actual email account with it is now gone.

If the login was compromised, just change the user’s password, thus disabling them from abusing your server, while keeping website and stuff active.

Firewalls don’t help against compromised email accounts or websites. Still it doesn’t hurt of course to only have those ports open you really need. I do the same with an external (virtual) firewall on my systems.

Using a software like OSSEC is also advisable. An alternative (which I myself use) is CSF/LFD (ConfigServer Security & Firewall / Login Failure Daemon), which integrates nicely with Webmin.

TSC Thanks for your help. I reset the firewall as per your instructions. I think it may have not even been on before, oops!

I installed ossec, but I see it is not monitoring the mail log. I had a brute force attack that obviously succeeded at some point. Will ossec block this in the future?

I will post a bit of the mail log to see if anyone has any further advice.

here is the begining of the current log

Dec 22 07:35:38 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:35:41 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:35:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:35:58 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:36:01 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:36:04 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:36:18 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:36:22 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:36:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:36:38 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:36:42 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:36:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:36:58 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:37:03 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:37:04 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:37:18 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:37:23 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:37:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:37:39 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:37:43 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:37:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:37:59 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:38:03 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:38:04 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:38:19 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:38:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:38:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:38:39 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:38:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48 Dec 22 07:38:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38 Dec 22 07:38:59 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39 Dec 22 07:39:03 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38

You can see the brute forceness of it - this goes on untill the next bit I will post. This is somewhere around the time the mail started being sent from the server, and was shortly blacklisted.

Dec 24 08:10:09 sdc postfix/pickup[10970]: 6E6D584032: uid=33 from= Dec 24 08:10:09 sdc postfix/cleanup[13860]: 6E6D584032: message-id=<20131224121009.6E6D584032@sdc.starlingdesign.ca> Dec 24 08:10:09 sdc postfix/qmgr[1789]: 6E6D584032: from=, size=7288, nrcpt=1 (queue active) Dec 24 08:10:09 sdc postfix/local[13862]: 6E6D584032: to=, orig_to=, relay=local, delay=0.14, delays=0.04/0.01/0/0.09, dsn=2.0.0, status=$ Dec 24 08:10:09 sdc postfix/qmgr[1789]: 6E6D584032: removed Dec 24 08:10:15 sdc postfix/anvil[13465]: statistics: max connection rate 1/60s for (smtp:192.254.250.167) at Dec 24 08:06:54 Dec 24 08:10:15 sdc postfix/anvil[13465]: statistics: max connection count 1 for (smtp:192.254.250.167) at Dec 24 08:06:54 Dec 24 08:10:15 sdc postfix/anvil[13465]: statistics: max cache size 1 at Dec 24 08:06:54 Dec 24 08:20:12 sdc postfix/pickup[10970]: 2BA4684032: uid=33 from= Dec 24 08:20:12 sdc postfix/cleanup[15506]: 2BA4684032: message-id=<20131224122012.2BA4684032@sdc.starlingdesign.ca> Dec 24 08:20:12 sdc postfix/qmgr[1789]: 2BA4684032: from=, size=7288, nrcpt=1 (queue active) Dec 24 08:20:13 sdc postfix/local[15509]: 2BA4684032: to=, orig_to=, relay=local, delay=1.7, delays=0.02/0.01/0/1.7, dsn=2.0.0, status=se$ Dec 24 08:20:13 sdc postfix/qmgr[1789]: 2BA4684032: removed Dec 24 08:27:26 sdc postfix/pickup[10970]: 1F6EB84032: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 1F6EB84032: message-id=<20131224122726.1F6EB84032@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 1F6EB84032: from=, size=3463, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 263A584033: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 263A584033: message-id=<20131224122726.263A584033@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 263A584033: from=, size=3500, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 2E80A84034: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 2E80A84034: message-id=<20131224122726.2E80A84034@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 2E80A84034: from=, size=3484, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 3D61684036: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 3D61684036: message-id=<20131224122726.3D61684036@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 3D61684036: from=, size=3483, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 458C984037: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 458C984037: message-id=<20131224122726.458C984037@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 458C984037: from=, size=3475, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 4D67384038: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 4D67384038: message-id=<20131224122726.4D67384038@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 4D67384038: from=, size=3499, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 5591B84039: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 5591B84039: message-id=<20131224122726.5591B84039@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 5591B84039: from=, size=3484, nrcpt=1 (queue active) Dec 24 08:27:26 sdc postfix/pickup[10970]: 5D2168403A: uid=1014 from= Dec 24 08:27:26 sdc postfix/cleanup[16573]: 5D2168403A: message-id=<20131224122726.5D2168403A@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 5D2168403A: from=, size=3503, nrcpt=1 (queue active)

There are lots of different patterns of what was going on. I am really not sure what to look for.

here is after I was blacklisted, and the mail started to bounce

Dec 24 08:30:52 sdc postfix/qmgr[1789]: 26E068405A: from=<>, size=5569, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/pickup[17463]: 34FBF84064: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 34FBF84064: message-id=20131224123052.34FBF84064@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 34FBF84064: from=ourshore.ca@sdc.starlingdesign.ca, size=3494, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16607]: D03868407B: to=larunmusic@yahoo.com, relay=mta5.am0.yahoodns.net[66.196.118.33]:25, delay=0.41, delays=0/0/0.18/0.23, dsn=5.0.0,$
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 41F8B84084: message-id=20131224123052.41F8B84084@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/pickup[17463]: 426D684085: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 426D684085: message-id=20131224123052.426D684085@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 41F8B84084: from=<>, size=5651, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 426D684085: from=ourshore.ca@sdc.starlingdesign.ca, size=3512, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/bounce[16677]: D03868407B: sender non-delivery notification: 41F8B84084
Dec 24 08:30:52 sdc postfix/qmgr[1789]: D03868407B: removed
Dec 24 08:30:52 sdc postfix/pickup[17463]: 4D6DA8407B: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 4D6DA8407B: message-id=20131224123052.4D6DA8407B@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 4D6DA8407B: from=ourshore.ca@sdc.starlingdesign.ca, size=3501, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16578]: 7E48C8405B: to=lastflightout777@wmconnect.com, relay=mailin-02.mx.aol.com[64.12.88.164]:25, delay=0.85, delays=0.01/0/0.34/0.5, $
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 5E5B884086: message-id=20131224123052.5E5B884086@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 5E5B884086: from=<>, size=5543, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/bounce[16678]: 7E48C8405B: sender non-delivery notification: 5E5B884086
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 7E48C8405B: removed
Dec 24 08:30:52 sdc postfix/pickup[17463]: 5F55D8405B: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 5F55D8405B: message-id=20131224123052.5F55D8405B@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/smtp[16586]: E681F84041: to=lastlightout777@wmconnect.com, relay=mailin-04.mx.aol.com[64.12.88.131]:25, delay=0.49, delays=0.04/0.01/0.34/0.1$
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 5F55D8405B: from=ourshore.ca@sdc.starlingdesign.ca, size=3505, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16591]: B816084076: to=lathen_flick@hotmail.com, relay=mx4.hotmail.com[65.55.37.72]:25, delay=0.67, delays=0.01/0/0.25/0.41, dsn=2.0.0, $
Dec 24 08:30:52 sdc postfix/qmgr[1789]: B816084076: removed
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 6927084083: message-id=20131224123052.6927084083@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/bounce[17188]: E681F84041: sender non-delivery notification: 6927084083
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 6927084083: from=<>, size=5589, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/qmgr[1789]: E681F84041: removed
Dec 24 08:30:52 sdc postfix/smtp[16588]: DB5408407D: to=lathena_laddaran@hotmail.com, relay=mx2.hotmail.com[65.54.188.126]:25, delay=0.58, delays=0.06/0/0.23/0.28, dsn=2$
Dec 24 08:30:52 sdc postfix/qmgr[1789]: DB5408407D: removed
Dec 24 08:30:52 sdc postfix/pickup[17463]: 7313184041: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 7313184041: message-id=20131224123052.7313184041@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 7313184041: from=ourshore.ca@sdc.starlingdesign.ca, size=3501, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/pickup[17463]: 7D4EE84076: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 7D4EE84076: message-id=20131224123052.7D4EE84076@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 7D4EE84076: from=ourshore.ca@sdc.starlingdesign.ca, size=3496, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16619]: C51078407A: to=lastlap02@wmconnect.com, relay=mailin-04.mx.aol.com[64.12.138.161]:25, delay=0.83, delays=0/0/0.67/0.15, dsn=5.1.$
Dec 24 08:30:52 sdc postfix/pickup[17463]: 99FD684087: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 99FD684087: message-id=20131224123052.99FD684087@sdc.starlingdesign.ca
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 99FD684087: from=ourshore.ca@sdc.starlingdesign.ca, size=3503, nrcpt=1 (queue active)

here this was yesterday, I thought it was strange as it went from POP3 to a postfix/smtpd

Dec 23 02:28:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:28:56 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:29:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:29:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:29:15 sdc postfix/smtpd[15309]: connect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:15 sdc postfix/smtpd[15307]: connect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:15 sdc postfix/smtpd[15310]: connect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:17 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:29:18 sdc postfix/smtpd[15309]: warning: h1832461.stratoserver.net[85.214.85.40]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:29:18 sdc postfix/smtpd[15310]: warning: h1832461.stratoserver.net[85.214.85.40]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:29:18 sdc postfix/smtpd[15307]: warning: h1832461.stratoserver.net[85.214.85.40]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:29:18 sdc postfix/smtpd[15309]: lost connection after AUTH from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15309]: disconnect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15310]: lost connection after AUTH from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15310]: disconnect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15307]: lost connection after AUTH from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15307]: disconnect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:29:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:29:37 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:29:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:29:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:29:57 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:30:00 sdc postfix/smtpd[15309]: connect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:00 sdc postfix/smtpd[15307]: connect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:00 sdc postfix/smtpd[15310]: connect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15307]: warning: h1960653.stratoserver.net[85.214.84.44]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:30:03 sdc postfix/smtpd[15310]: warning: h1960653.stratoserver.net[85.214.84.44]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:30:03 sdc postfix/smtpd[15309]: warning: h1960653.stratoserver.net[85.214.84.44]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:30:03 sdc postfix/smtpd[15309]: lost connection after AUTH from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15309]: disconnect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15307]: lost connection after AUTH from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15307]: disconnect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15310]: lost connection after AUTH from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15310]: disconnect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:30:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:30:17 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:30:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:30:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:30:37 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:30:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38

Thanks for the help so far and Merry Christmas. (I can’t believe I am dealing with this garbage now.)

Not sure if these will help at all, but it’s the graphs from Munin re: postfix.

You can see the backlog starts around 08:00 today.

How do I disable for this one? All the email seems to be being sent from ourshore.ca

The actual email being sent is coming from manager@ourshore.ca (that email and user does not exist)

here is the full raw message w/headers from a bounced mail
Return-Path: <>
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
sdc.starlingdesign.ca
X-Spam-Level:
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS
autolearn=ham version=3.3.2
X-Original-To: ourshore.ca@sdc.starlingdesign.ca
Delivered-To: ourshore.ca@sdc.starlingdesign.ca
Received: by sdc.starlingdesign.ca (Postfix)
id AC35084075; Tue, 24 Dec 2013 16:19:43 -0400 (AST)
Date: Tue, 24 Dec 2013 16:19:43 -0400 (AST)
From: MAILER-DAEMON@sdc.starlingdesign.ca (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: ourshore.ca@sdc.starlingdesign.ca
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary=“1C71084055.1387916383/sdc.starlingdesign.ca”
Message-Id: 20131224201943.AC35084075@sdc.starlingdesign.ca

This is a MIME-encapsulated message.

–1C71084055.1387916383/sdc.starlingdesign.ca
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host sdc.starlingdesign.ca.

I’m sorry to have to inform you that your message could not
be delivered to one or more recipients. It’s attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

               The mail system

msosinski@woh.rr.com: host cdptpa-pub-iedge-vip.email.rr.com[107.14.166.70]
said: 554 Invalid recipient (in reply to RCPT TO command)

–1C71084055.1387916383/sdc.starlingdesign.ca
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; sdc.starlingdesign.ca
X-Postfix-Queue-ID: 1C71084055
X-Postfix-Sender: rfc822; ourshore.ca@sdc.starlingdesign.ca
Arrival-Date: Tue, 24 Dec 2013 16:19:15 -0400 (AST)

Final-Recipient: rfc822; msosinski@woh.rr.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; cdptpa-pub-iedge-vip.email.rr.com
Diagnostic-Code: smtp; 554 Invalid recipient

–1C71084055.1387916383/sdc.starlingdesign.ca
Content-Description: Undelivered Message
Content-Type: message/rfc822

Return-Path: ourshore.ca@sdc.starlingdesign.ca
Received: by sdc.starlingdesign.ca (Postfix, from userid 1014)
id 1C71084055; Tue, 24 Dec 2013 16:19:15 -0400 (AST)
To: msosinski@woh.rr.com
Subject: Delivery Canceling
X-PHP-Originating-Script: 1014:sys09725848.php
From: “Costco Shipping Manager” manager@ourshore.ca
X-Mailer: rajahmadsen
Reply-To: “Costco Shipping Manager” manager@ourshore.ca
Mime-Version: 1.0
Content-Type: multipart/alternative;boundary="----------138791635552B9EC431A6AA"
Message-Id: 20131224201943.1C71084055@sdc.starlingdesign.ca
Date: Tue, 24 Dec 2013 16:19:15 -0400 (AST)

------------138791635552B9EC431A6AA
Content-Type: text/plain; charset=“ISO-8859-1”; format=flowed
Content-Transfer-Encoding: 7bit

 

Costco

       WHOLESALE

 
 
 

Unfortunately the delivery of your order
COS-0031180544
was cancelled since
the specified address of the recipient was not correct. You are
recommended to complete

this form and send it
back with your reply to us.

Please do this within the period of one week - if we dont get your
timely reply you will be paid your money back less 21% since your
order was booked for Christmas.

 

1998 -
2013
Costco Wholesale Corporation
All rights reserved

------------138791635552B9EC431A6AA
Content-Type: text/html; charset=“ISO-8859-1”;
Content-Transfer-Encoding: 7bit

 
Costco
       WHOLESALE
     

Unfortunately the delivery of your order COS-0031180544 was cancelled since the specified address of the recipient was not correct. You are recommended to complete this form and send it back with your reply to us.

Please do this within the period of one week - if we dont get your timely reply you will be paid your money back less 21% since your order was booked for Christmas.
 

1998 - 2013
Costco Wholesale Corporation
All rights reserved

------------138791635552B9EC431A6AA–

–1C71084055.1387916383/sdc.starlingdesign.ca–

Like TSC said, brute force dictionary attacks are very common. That’s what software like OSSEC or LFD are for: Detecting these and (temp/perm) blocking the attacking IP.

These repeating log lines look interesting:

Dec 24 08:27:26 sdc postfix/pickup[10970]: 1F6EB84032: uid=1014 from=<ourshore.ca> Dec 24 08:27:26 sdc postfix/cleanup[16573]: 1F6EB84032: message-id=<20131224122726.1F6EB84032@sdc.starlingdesign.ca> Dec 24 08:27:26 sdc postfix/qmgr[1789]: 1F6EB84032: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3463, nrcpt=1 (queue active)

They suggest that the spam is not delivered to your system via SMTP, but probably from a local website. The user ID 1014 is being used to put it in the Postfix queue. You might want to check in /etc/passwd which user that is, and check if their website has been compromised.

use 1014 is ourshore.ca - it was created by virtualmin when I created the virtual server ourshore.ca

I changed it’s password via usermin, but that didn’t stop the mail.

What should I do? Can I just comment out the line in /etc/passwd for now?

I noticed there is also a /etc/passwd file, the difference being it contained the new ossec user. That has me stumped too.

What is my next move?

Changing the password for that user won’t have an effect if there’s malicious code in their website. It’s probable that a hacker found a security hole in the web software and installed stuff there. Access to that is independent from the administrative user. You need to disable the virtual server, thus making the web files inaccessible. Also check that user’s Apache logfiles for suspicious activity, like downloads or uploads of unknown files. Do the same with ProFTPD logs.

A changed passwd file is normal if a user (here: OSSEC) was added. It’s to be expected, since you installed OSSEC.

You might want to run the software “Linux Malware Detect” on the potentially compromised web directory; best is if you run it on the entire /home. Also check out what kind of web software the “ourshore.ca” is using; make sure it’s up to date and is not using a version with known security issues.

Oh, also check for active processes that are running under the user ID 1014, with ps aux | grep 1014. Maybe they started a process that’s sending the mail or is doing other stuff.

Thanks again for the help so far. My virtual-server ourshore.ca is disabled and has been scince yesterday. As soon as I noticed the spam I killed Postfix and stopped the actual sending of the spam. That was my #1 goal. Now I am just looking to repair and prevent, like any good internet citizen should :wink:

I do not use FTP at all, everything is done over SSH via SFTP etc. ProFTP is not even running, I think. I have unchecked it, and it does not list on the System Information -> status section. What might I do to make sure FTP is in fact off? (command line?)

The more I think about it, it is likely that the website ourshore.ca has some malicious file/thing on it. I just need to find it and remove it. But not today. For now, the Spam is not being sent, nor is it bothering my server. Everything else is running as per normal.

I installed fail2ban, but I don’t think it is working right as it only lists a jail for SSH.

I also now have the OSSEC installed and should be running, so thats good to. Should I remove fail2ban?

The server is running Ubuntu 12.04 LTS

ourshore.ca is built on Joomla 2.5 and is up to date, along with any modules/plugins it uses. The hard part is going to be figuring out how something got on there in the first place. I also have current backups, so I think I will take one more, then restore to a previous one, and do some file compare. . . but not till later. I will deal with this some more when I “go back to work”

Thanks again so far and Merry Christmas!

ps aux | grep 1014 output

root 13219 0.0 0.1 9388 936 pts/0 S+ 13:29 0:00 grep --color=auto 1014

Good, so there are no processes running as ID 1014.

You should remove fail2ban, yes, it’ll probably lead to problems if you have multiple software installed that does login failure blocking.

If you have “ourshore.ca” disabled, you should be seeing no more spam mails being added to Postfix.

As I said, you might want to use “Linux Malware Detect” to scan your /home folder.

I can’t really start from scratch, but if I get nowhere I may just do that.

A question about OSSEC - It seems to be working like it should, but how can I tell what actions it has taken.

This morning I got an email alert about failed SSH login attempts, 2 sets of about 5 each from the same IP. I assume OSSEC blocked the IP, but how do I know? Where will it do this?

I have read through most of OSSEC’s website but didn’t find anything.