Security problem

Had a strange attack on a webmin server this weekend. Probably not a webmin problem, but just in case I thought I would report it.

Around 10PM Eastern a remote user at an AOL address attempted to log on once to one of my webmin servers via account "admin" and was rejected because of a bad password. Then the same IP attempted to log in as user "steve" and was successful on the first try.

This machine had only 3 webmin users defined - admin, steve, & a third name. None of the passwords were vulnerable to dictionary attack, although steve’s has only 5 characters long. All have been changed now.

By the time I was notified of the problem, the intruder had a few hours head start on me installing various root kits. Since this machine was supporting live content for 100’s of users, I had to decide to either pull the data plug or fight while still online. I choose the fight and battled for a few hours closing various ports and hunting down alien & subverted files. In the end, I own the race. In hindsight, after finding all that was installed, I should have pulled the plug right away. There were at least three time where I had just blocked an specific evil action and the intruder tried to do that same action within 5 minutes. If I had stopped for a cup of coffee, I would have lost everything on the machine.

In the end, he/she/they threw a nasty denial of service attack against the server, and every night since has unleashed a few nasty kiddy scrips against the same machine.

AOL was less then helpful, they didn’t even respond to my report.

I assume that we were the victim of some sort of social engineering attack, as there is no way they could have got the password in one try. We’re reviewing our internal procedures and controls. But, just in case others have had successful crack of user passwords in one attempt, it would indicate a security problem.

Hey Steve,

Harrowing tale. Glad to hear things are under control again.

It’s interesting that the attacker came in on a non-root account. Since “root” and/or “admin” is known to exist on any Webmin server (OK, not know…but definitely safe to assume), it’s surprising that a “steve” account would even be in the running of usernames to attempt. Since the attacker knew both the username and password, they were clearly working with privileged knowledge.

My best guess is they had, via some other attack vector, obtained your shadow password file. Then, the steve account having the shortest password, brute forced the password. John the Ripper, along with a big corpus, can crack even “strong” passwords very quickly if they are short enough–and 5 chars is definitely short enough. Brute force can’t really be the culprit in the case of ssh or Webmin logins.

There was, several revisions back, a local file exploit bug in Webmin and Usermin, which could explain someone having access to the shadow file–but that was fixed over a year ago, so you’d have to have a pretty old Webmin or Usermin to be exposed. There could, however, be some other package on your system that has a similar bug.

SteveAcup wrote:

AOL was less then helpful, they didn’t even respond to my report.

Oh yes, the famous AOL balk. THey would be the ones quickest to block your IP block as a perceived spammer, slowest to unblock it yet the least helpful in taking accountability for their own relays. AOL relays are the most popular to exploit and their RR subscribers are the happiest to leave their machines open.

As Joe was possibly alluding to, you may have been already compromised for some time already before this happened. THere’s been alot of activity over the last week and your machine might have been called up.

Early last year we found one of our boxes exploited and couldn’t find an entry point but found lots of exploits. Not until we set all the virtual servers on FCGId operating as the server owner did we isolate where it was coming from. Until then all the trouble was showing up as owned by apache, including running IRC and fileserver connections. THen we found that it was a website purchased on ebay. Apparently, hidden from view, all the content was tied up in a 144MB MySQL database to be called by index and arranged. Apparently scripts were in there waiting to be called up by the former server owner (explains why the perps kept getting the new password even after it was changed.

Like you experienced, when we started attacking the attackers they got really aggressive. It’s interesting to note that simply changing the domain name ( even just the TLD) was enough to stop the onslaught but we did remove the website just the same.

THis happened on a server we just started up while working VM’s bells and whistles. We’ve since tightened up PHP and in particular IP tables to stop brute force and haven’t had any real trouble since. We still have attempts to run remote scripts from remote PHP calls but they have become ineffective, just nuisance trolling.

You may want to add more characters to that password, when h@k3rz (hackers) use funny letters, there’s a reason.

hope that helps,
Dan

I didn’t see any response from Joe… Did I miss something?

All our passwords are multiple character types, not straight dictionary words or even just random letters. But we did not enforce minimum length, now we do.

When checking older logs, I did find an ancient brute force attack against user steve on ftp. Turns out FTP does not log successful logins in auth log, just the ones that fail. (Old version of FreeBSD). Newer versions are better. When Joe gets that Freebsd script out we’ll just upgrade and migrate everything.

Still doesn’t explain why the attacker tried “admin” first 1 time and then hit “steve”. Either they thought they had a password from before or they were just adding some misdirection.

Steve

OK… I can see Joe’s post now in a new window, but my 2nd post doesn’t show up. In a third window, Joe’s post is gone again but my 2nd post is there. Since my initial post looks like it double posted, there may be some pointer problems here.

Did I mention that they logged straight into webmin on our non-standard webmin port, not ssh or ftp? And no other accounts were touched yet, as far as I know. They did cat a system file with user names via the webmin command shell interface, but it was a file without passwords.

I’m still looking for signed I missed something. So far the box looks good, clear of rootkits, no unexplained activity, and no odd php files in any hosted domains or user space. Still not a 100% trusted system, but I’m hoping to transition it all to a new Freebsd box with VM pro soon.

Steve

Probably misdirection, why give up your fastball when you can get in throwing over the plate.

I didn’t realize you were running FreeBSD. I gave up on FreeBSD long ago due, in part, to their attitude and frankly the simpler CentOS. I had been running Slackware but it’s more hackware for guys who truly enjoy hacking around the code and it’s not enterprise worthy. I got frustrated with FreeBSD because they maintain an attitude of leaving critical services out of the base because if you are an admin who needs these they assume you are well versed enough in UNIX to install and administer the service. That and being really slow to keep up with new hardware drivers. I’m also sales and support so I really don’t have the time to be a network hyperguru. So I won’t be much FreeBSD help.

You might want to check the domain access logs and look for a preponderance of calls to remote scripts. Typically if these guys gain access they don’t damage the source. THey’ll map the server then post it on the hacker websites with the domain name that had access, apparently. Changing the IP didn’t help the cause, but when we changed the domain name the activity pretty much stopped. I’d found myself giving the hackers way too much credit up until I’d realized their access was domain name dependent.

When I’d mentioned the IRC and fileserver points I’d meant to check the processes. They don’t come through any security points so you won’t find them in an access log outside of Apache. They come in with an http call to a remote file to establish persistent connections. We had as many as 50 concurrent connections as IRC or fileserver just sitting there not loading the CPU. I’d kill the process and find it back when the list refreshed. Like I said, we haven’t had any problems since changing the domain name and eventually just getting rid of the website.

A side note on the forum posts. If you’re taking a while to type your post, it may time out and you’ll get hit with the repost data prompt, just say cancel if that comes up ( you may want to select all and copy your post before sending if you’ve been burned) otherwise it will double post your topic and then replyies get kindof lost in the shuffle.

Dan

Takes me a couple of tries to see all the responses now because of the original double post. But I’ll keep up…

FreeBSD does lag a bit with drivers, but overall I’m fairly happy with their security & performance. I don’t mind installing the services needed.

I’ve been checking the logs of course, but my monitoring has been with tcpdump on the gateway router to watch all traffic to and from the compromised server. And we have NIDS devices monitoring. Any packets I cannot explain raise a flag and I track down the source. Haven’t had any unexplained traffic since about 1/2 way through the first day.

There could be something hidden, but we’ll be transitioning to a new server soon anyway.