This may be a true apache issue, but I am posting here because apache was configured with Virtualmin GPL and I’m sure this issue may be a problem for someone in the future. The server has 3 virtual hosts. Proxying is turned on for each of them and they are pointing to internal application servers. I use Virtualmin for the front end. I took these steps to configure:
1- Enable mod_proxy on server
2- Edit proxy website on Virtualmin server, tick off "Proxying enabled" and enter the internal server http://example01/
Now the problem: One of these virtual hosts is getting hammered by proxy requests. Enough to crash the application server several times a day (not the Virtualmin server, but the server it is proxying to). I don’t understand how the attackers are using my server as a proxy because I cannot configure it to be one. They must know some trick. I also don’t understand why out of 3 virtual hosts, only one is being attacked. Here is a clip of the log:
Here is the proxy.conf (no changes but here it is for proof):
$ cat /etc/apache2/mods-enabled/proxy.conf
<IfModule mod_proxy.c>
#turning ProxyRequests on and allowing proxying from all may allow
#spammers to use your proxy to send email.
ProxyRequests Off
<Proxy *>
AddDefaultCharset off
Order deny,allow
Deny from all
</Proxy>
# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
ProxyVia On
</IfModule>
Here is the virtual host conf:
$ cat example01.org.conf
<VirtualHost 192.168.3.56:80>
SuexecUserGroup "#1001" "#1002"
ServerName example01.org
ServerAlias www.example01.org
ServerAlias webmail.example01.org
ServerAlias admin.example01.org
ServerAlias lists.example01.org
DocumentRoot /home/vhost1/domains/example01.org/public_html
ErrorLog /var/log/virtualmin/example01.org_error_log
CustomLog /var/log/virtualmin/example01.org_access_log combined
ScriptAlias /cgi-bin/ /home/vhost1/domains/example01.org/cgi-bin/
ScriptAlias /awstats /home/vhost1/domains/example01.org/cgi-bin
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/vhost1/domains/example01.org/public_html>
Options -Indexes IncludesNOEXEC FollowSymLinks
allow from all
AllowOverride All
</Directory>
<Directory /home/vhost1/domains/example01.org/cgi-bin>
allow from all
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.example01.org
RewriteRule ^(.*) https://example01.org:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.example01.org
RewriteRule ^(.*) https://example01.org:10000/ [R]
Alias /dav /home/vhost1/domains/example01.org/public_html
Alias /pipermail /var/lib/mailman/archives/public
<Location /dav>
DAV On
AuthType Basic
AuthName example01.org
AuthUserFile /home/vhost1/domains/example01.org/etc/dav.digest.passwd
Require valid-user
ForceType text/plain
Satisfy All
</Location>
<Files awstats.pl>
AuthName "example01.org statistics"
AuthType Basic
AuthUserFile /home/vhost1/domains/example01.org/.awstats-htpasswd
require valid-user
</Files>
RedirectMatch /cgi-bin/mailman/([^/]*)(.*) https://example01.org:10000/virtualmin-mailman/unauthenticated/$1.cgi$2
RedirectMatch /mailman/([^/]*)(.*) https://example01.org:10000/virtualmin-mailman/unauthenticated/$1.cgi$2
ProxyPreserveHost On
ProxyPass / http://example01/example01/
ProxyPassReverse / http://example01/example01/
<Proxy *>
allow from all
</Proxy>
</VirtualHost>
What exactly is the problem? (It sounds like you’re saying proxying is a problem…but you’ve turned on proxying to that internal server, and proxying through to it should be exactly what it does.)
Are you saying that your internal server is behaving as an open proxy on behalf of users?
Perhaps if you explained which of the hosts in that log file are yours and which are not (if my open proxy theory is actually what you’re asking about), we’d be able to understand a bit more about what is happening here. On first glance I assumed everything in the log was your servers…but on a third reading it sounds like you’re a bit too panicked for that to be the case.
Yes I think I am running an open proxy. Or possibly being/have been exploited. The proxying works as it should for my purposes, but it seems to have opened up to the world rather than just the internal network. This server is being hammered by multiple hosts with funny looking GET requests. I can’t say for sure if it is actually being used successfully, but what I can say is that my IP was definitely tagged as a possible open proxy by some evil nework. The log posted is a sample of the bad traffic that I am getting in the log. None of those have to do with me or my network. I’m not running any ad services.
Also, I tried testing it out the only way I know how. Enter my IP and port 80 into Firefox network proxy settings. It does not work. I can’t figure out how they are using it as a proxy (if they are at all)
I don’t see anything that would open up Apache for use as a proxy in that configuration. You might check to be sure you don’t have ProxyRequests set to on anywhere else in the Apache configuration:
grep -r ProxyRequests /etc/apache2/*
And, you might also turn off ProxyVia, as it is normally only used in Forward Proxy situations, and might be contributing to the problem somehow.
While we figure this out, if there are recurring themes to the sites being proxied to, you could add ProxyBlock rules for those specific sites.
There are no additional ProxyRequests other than the one in proxy.conf which is turned off, so that looks good
I have turned off ProxyVia and it has not stopped the requests.
This has been going on for a few days so I have a very good number of hosts to block, I’ll be doing that next as a band-aid until I get this figured out.
Here is another clip of the log. What do you make of all the 404s?
Hey Sauce,
Did you ever find out what the underlying issue was? I have an almost identical issue (judging by the logfiles) that just cropped up right around the exact time yours did. The only variant is that one of the servers is running Tomcat (apache variant) and the other is running an IIS server with ColdFusion. These were vendor setup, but we did a google search on our domain name and +proxy, and we were all over the place. This is with us having the PIX setup to only allow port 80 and 443 traffic through the conduit. Both of these boxes are in the internal network, but have translations through the PIX to them. At one point, we had over 2800 active connections on one box and 2400 on the other. It was bringing our 45mb pipe to its knees.
Our logs look pretty much identical, and the only way we’ve managed to even throttle some of the traffic is to force everything through SSL, and block port 80 - which is not really a true fix. I apologize for posting on these forums, but I’ve yet to see much other information or suggestions elsewhere on the web - we’ve been combating this for a few weeks now. Any suggestions at this point would be greatly appreciated.
This forum was down for a few days so I wasn’t able to post back.
Joe noticed my "ProxyVia" got turned on in proxy.conf. Default is off. I probably turned it on by accident. Once I turned it off, my problems were solved.