Server hacked, how can I stop it?

So I’m pretty sure I’ve been hacked in some manner but I’m too much of a noob to know how it happened or how to stop it. It came to my attention because the bandwidth on my VPS has been way over quota by 2x and from what I can tell it seems like it came from a hack located at this site, which itself seems to have been hacked to hold that file.

http://alisonmcleastudio.com/jpg/wpa.bwe
that may need to be hidden so others don’t use it or something

here is the results of the error log located at /var/log/httpd/errorlog, which is line after line after line of this type of stuff:

–2013-12-09 10:13:02-- http://alisonmcleastudio.com/jpg/wap.bwe
Resolving alisonmcleastudio.com… 50.63.101.1
Connecting to alisonmcleastudio.com|50.63.101.1|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 40802 (40K)
Saving to: `wap.bwe’

 0K .......... .......... .......... .........            100% 17.4M=0.002s

2013-12-09 10:13:02 (17.4 MB/s) - `wap.bwe’ saved [40802/40802]

[Mon Dec 09 10:13:39 2013] [warn] mod_fcgid: process 5739 graceful kill fail, sending SIGKILL
rm: cannot remove *alisonmcle*': No such file or directory --2013-12-09 10:15:42-- http://alisonmcleastudio.com/jpg/wpa.bwe Resolving alisonmcleastudio.com... 50.63.101.1 Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 40803 (40K) Saving to: wpa.bwe’

 0K .......... .......... .......... .........            100% 5.87M=0.007s

2013-12-09 10:15:42 (5.87 MB/s) - `wpa.bwe’ saved [40803/40803]

–2013-12-09 10:15:42-- http://alisonmcleastudio.com/jpg/wap.bwe
Resolving alisonmcleastudio.com… 50.63.101.1
Connecting to alisonmcleastudio.com|50.63.101.1|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 40802 (40K)
Saving to: `wap.bwe’

 0K .......... .......... .......... .........            100% 12.1M=0.003s

2013-12-09 10:15:42 (12.1 MB/s) - `wap.bwe’ saved [40802/40802]

[Mon Dec 09 10:16:19 2013] [warn] mod_fcgid: process 5743 graceful kill fail, sending SIGKILL
rm: cannot remove *alisonmcle*': No such file or directory --2013-12-09 10:18:51-- http://alisonmcleastudio.com/jpg/wpa.bwe Resolving alisonmcleastudio.com... 50.63.101.1 Connecting to alisonmcleastudio.com|50.63.101.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 40803 (40K) Saving to: wpa.bwe’

 0K .......... .......... .......... .........            100% 16.7M=0.002s

2013-12-09 10:18:51 (16.7 MB/s) - `wpa.bwe’ saved [40803/40803]

–2013-12-09 10:18:51-- http://alisonmcleastudio.com/jpg/wap.bwe
Resolving alisonmcleastudio.com… 50.63.101.1
Connecting to alisonmcleastudio.com|50.63.101.1|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 40802 (40K)
Saving to: `wap.bwe’

 0K .......... .......... .......... .........            100% 18.0M=0.002s

2013-12-09 10:18:51 (18.0 MB/s) - `wap.bwe’ saved [40802/40802]

Does anyone know exactly what this hack is doing, and how can I stop it?

Thanks

Howdy,

It sounds like you’ve done a lot of the hard part – determining which site/account is the source of the problem.

It’s not simple to determine what the hack is doing – but I’ll offer that whatever it is, we don’t want it doing that :slight_smile:

The first thing I’d suggest doing is reviewing all the software installed onto that account, and making sure it’s fully up to date. If it’s using WordPress, for example, make sure that it’s running the most recent WordPress version.

You’d also want to look for a file named “wpa.bwe”, possibly in the “jpg” directory of that domain.

One thing you may want to verify is to see if there’s any email in the mail queue, as some breakins send spam.

You can determine that by running this command:

mailq | tail -1

You may also want to see if there are any processes running as this user, and if so, you may want to kill them.

-Eric

Hi Eric,

The domain I mentioned is not associated with my server at all, that’s on a foreign site, but somehow my server is connecting to it and repeatedly downloading files, though it doesn’t look like they get stored anywhere. However, the hack is eating up tons of bandwidth and it’s going to cost me a lot of money to cover the overages if I can’t get this figured out quickly.

I ran the mail command you gave and it doesn’t appear that any mail is being sent, so I’m not sure exactly what all the bandwidth is being used for.

Thanks

Hmm, normally, log messages go into the individual domain’s error logs, located in $HOME/logs/error_log.

That may mean that there’s some sort of web app setup on a default VirtualHost on your server.

Do you see any files in “/var/www/html/”?

-Eric

Same here. Exact same domain, same IP. I block the domain 50.63.101.1. But that’s not the perfect solution. Someone else who can help out>

You might first want to use tools like “iftop”, “netstat -tpn”, “atop” to find out which process is doing the requests and which user it is doing them as. From there we can progress further.

It is the /sbin/syslogd process. Done by the Root user.

[code]
This is an automated message notifying you that the 5 minute load average on your system is 14.54.
This has exceeded the 10 threshold.

One Minute - 21.61
Five Minutes - 14.54
Fifteen Minutes - 9.27

top - 19:20:04 up 11:29, 0 users, load average: 21.88, 14.71, 9.36
Tasks: 202 total, 9 running, 189 sleeping, 0 stopped, 4 zombie
Cpu(s): 1.3%us, 1.3%sy, 0.0%ni, 95.9%id, 0.4%wa, 0.0%hi, 0.0%si, 1.0%st
Mem: 3922608k total, 2033336k used, 1889272k free, 288964k buffers
Swap: 1048564k total, 0k used, 1048564k free, 727096k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21635 apache 20 0 137m 6920 1072 R 61.5 0.2 0:19.29 /sbin/syslogd
21646 apache 20 0 137m 6920 1068 R 61.5 0.2 0:20.92 /sbin/syslogd
21648 apache 20 0 137m 6924 1068 R 61.5 0.2 0:23.88 /sbin/syslogd
21651 apache 20 0 137m 6924 1072 R 61.5 0.2 0:27.79 /sbin/syslogd
21658 apache 20 0 137m 6916 1068 R 61.5 0.2 0:19.53 /sbin/syslogd
21659 apache 20 0 137m 6924 1068 R 61.5 0.2 0:20.52 /sbin/syslogd
21650 apache 20 0 137m 6924 1072 R 37.2 0.2 0:19.58 /sbin/syslogd
21662 apache 20 0 137m 6920 1068 R 35.9 0.2 0:19.01 /sbin/syslogd
3308 apache 20 0 137m 6948 1096 D 1.3 0.2 0:02.60 /sbin/syslogd
7384 apache 20 0 132m 6788 1080 S 1.3 0.2 0:37.13 /sbin/syslogd
7464 apache 20 0 132m 6784 1080 S 1.3 0.2 0:36.55 /sbin/syslogd
27484 root 20 0 15024 1260 880 R 1.3 0.0 0:00.01 /usr/bin/top -c -b -n 1
1 root 20 0 19232 1492 1220 S 0.0 0.0 0:00.82 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd] [/code]
As you can see in the script from the startpost; http://alisonmcleastudio.com/jpg/wpa.bwe > there is the /sbin/syslogd name defined

The “syslogd” name might be a fake done by an exploit, or the executable itself is an exploit. According to your process list, syslogd is run by apache, not root. And I doubt the syslog daemon would run as apache.

I didn’t get your last line. What is “defined” where?

Oh, I see it now. Yeah this “wpa.bwe” seems to be a bot (PERL script) that hides itself as “/sbin/syslogd” and participates in IRC activity (connects to a CNC I suppose). Well I suppose someone exploited a weakness in some web software and had Apache download and run multiple copies of that script.

First step would be killing these processes, rename (not delete, keep it for analysis) the script and review all software installed in your web sites. Software like “Linux Malware Detect” can help with that. Make sure all web software is up-to-date. Remove any insecure software or plugins.

Review logs in /var/log for suspicious activity, also do a “last” which shows logins. Review CRON job tables. Change passwords of web/email users that might be compromised. Change the root password for good measure. Use rootkit detection software to see if there’s a deeper reaching compromise.

The file “wpa.bwe” might be related to Wordpress, maybe a known vulnerability/exploit. Make sure that all your Wordpress installations are up-to-date.

http://pastebin.com/9Nj8Pgv5 (found via Google, this is not a paste from me)

So it definitely looks like it’s some exploit related to one of my wordpress installations, I’m not sure if it’s a plugin or a theme issue though.

The plugins I have installed are all very popular and well known plugins like All in One SEO Pack and Form Maker. I know that forms are a particular vulnerability but I think that that plugin is pretty secure.

So that leaves a possible vulnerability with some of the theme files. I use the SAICO theme on this particular site, which did have some exploits when it was released but were fixed with an update later, though the developer has stopped supporting it. I think I’ll see about changing the theme and seeing if that does the trick. Unfortunately it does require me to redesign the site quite a lot.

Below is an image of the hack process /sbin/syslogs in the webmin process details. It says it’s a child process of “init” is that something I need to go root around for more so that it doesn’t automatically start up when I restart the server?

Also, how can I determine the IP address that the attack is coming from and just block that IP as well?

Also, I went back and checked in /var/logs/httpd/error_log and found that apache is connecting to a different thing now. This deal was downloading a new index.php and some supporting files into my wordpress installation and putting up a weird “dark web” site.

These are the log files, it did this same process several times over so I just copied one cycle of it. It saved the files to wp-content/uploads/2013/04

I ran a last command and no one else has logged into the server directly so it seems all the attacks are through the one website. The rogue processes are all running as the user for that particular domain and not as root or anything that indicates the entire server is compromised.

I still don’t understand exactly how the site is compromised but I updated wordpress to the newest 3.8 version and updated all plugins.

–2013-12-18 11:47:58-- http://yanaselandscaping.com/blog/wp-content/themes/echea/js/cufon-fonts/uploaded/header.tot
Resolving yanaselandscaping.com… 72.41.38.176
Connecting to yanaselandscaping.com|72.41.38.176|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 273165 (267K) [image/jpeg]
Saving to: `header.tot’

 0K .......... .......... .......... .......... .......... 18%  264K 1s
50K .......... .......... .......... .......... .......... 37%  792K 0s

100K … … … … … 56% 790K 0s
150K … … … … … 74% 742K 0s
200K … … … … … 93% 28.7M 0s
250K … … 100% 501K=0.4s

2013-12-18 11:47:59 (638 KB/s) - `header.tot’ saved [273165/273165]

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 266k 100 266k 0 0 468k 0 --:–:-- --:–:-- --:–:-- 528k
sh: /usr/bin/xdg-open: No such file or directory
–2013-12-18 11:48:01-- http://yanaselandscaping.com/blog/wp-content/themes/echea/js/cufon-fonts/uploaded/logo.tot
Resolving yanaselandscaping.com… 72.41.38.176
Connecting to yanaselandscaping.com|72.41.38.176|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 43216 (42K) [image/jpeg]
Saving to: `logo.tot’

 0K .......... .......... .......... .......... ..        100%  217K=0.2s

2013-12-18 11:48:01 (217 KB/s) - `logo.tot’ saved [43216/43216]

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
100 43216 100 43216 0 0 133k 0 --:–:-- --:–:-- --:–:-- 167k
sh: /usr/bin/xdg-open: No such file or directory
–2013-12-18 11:48:02-- http://yanaselandscaping.com/blog/wp-content/themes/echea/js/cufon-fonts/uploaded/tot.tot
Resolving yanaselandscaping.com… 72.41.38.176
Connecting to yanaselandscaping.com|72.41.38.176|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 3841 (3.8K) [text/html]
Saving to: `tot.tot’

 0K ...                                                   100%  302M=0s

2013-12-18 11:48:02 (302 MB/s) - `tot.tot’ saved [3841/3841]

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0
101 3841 101 3841 0 0 30252 0 --:–:-- --:–:-- --:–:-- 60968

Yes, your processes screenshot shows the Perl script running as a server owner, so you know which website is the culprit.

You might want to run “Linux Malware Detect” on the web directory in question, maybe it finds something.

Also change all passwords for the Wordpress in question, change the “admin” username to something not easily guessable, and put .htaccess protection on the backend directory.

So I ran Linux Malware Detect and the scan file reads thusly:

malware detect scan report for ip-50-62-213-134.ip.secureserver.net:
SCAN ID: 121913-1211.24547
TIME: Dec 19 12:12:14 -0700
PATH: /home/antiquemoney
TOTAL FILES: 30432
TOTAL HITS: 1
TOTAL CLEANED: 0

NOTE: quarantine is disabled! set quar_hits=1 in conf.maldet or to quarantine results run: maldet -q 121913-1211.24547
FILE HIT LIST:
{HEX}gzbase64.inject.unclassed.14 : /tmp/maldetect-1.4.2/files/clean/gzbase64.inject.unclassed

Linux Malware Detect v1.4.2 < proj@rfxn.com >

I did a grep in that directory for “base64_decode(” and got the following hits, all of which seem to have legitimate uses, not doing the normal base64 decode hack stuff I’ve seen before. Anymore ideas as to what may be the particular vulnerabilities that allowed the site to be hacked?

/home/antiquemoney/public_html/wp-includes/class-feed.php
/home/antiquemoney/public_html/wp-includes/ID3/module.audio.ogg.php
/home/antiquemoney/public_html/wp-includes/class-IXR.php
/home/antiquemoney/public_html/wp-includes/SimplePie/Sanitize.php
/home/antiquemoney/public_html/wp-includes/class-smtp.php
/home/antiquemoney/public_html/wp-admin/includes/file.php
/home/antiquemoney/public_html/wp-content/plugins/all-in-one-seo-pack/OAuth.php

Thanks for all the help so far