URGENT- Virtual host hammered by proxy requests

This may be a true apache issue, but I am posting here because apache was configured with Virtualmin GPL and I’m sure this issue may be a problem for someone in the future. The server has 3 virtual hosts. Proxying is turned on for each of them and they are pointing to internal application servers. I use Virtualmin for the front end. I took these steps to configure:

1- Enable mod_proxy on server
2- Edit proxy website on Virtualmin server, tick off "Proxying enabled" and enter the internal server http://example01/

Now the problem: One of these virtual hosts is getting hammered by proxy requests. Enough to crash the application server several times a day (not the Virtualmin server, but the server it is proxying to). I don’t understand how the attackers are using my server as a proxy because I cannot configure it to be one. They must know some trick. I also don’t understand why out of 3 virtual hosts, only one is being attacked. Here is a clip of the log:

221.224.78.227 - - [04/May/2009:14:31:12 +0000] "GET http://afe.specificclick.net/update.php HTTP/1.0" 302 5177 "http://www.autoefax.com/lease/rebates/make/land rover.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12" 221.224.78.248 - - [04/May/2009:14:31:12 +0000] "GET http://afe.specificclick.net/update.php HTTP/1.0" 302 5177 "http://www.autoefax.com/search/new/aston martin/vantage.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1" 125.66.112.149 - - [04/May/2009:14:31:13 +0000] "GET http://impgb.tradedoubler.com/imp?type(img)g(16427016)a(1079897)695742616 HTTP/1.0" 404 336 "http://www.ukbusiness100.com/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)" 221.224.78.245 - - [04/May/2009:14:31:13 +0000] "GET http://afe.specificclick.net/?l=1521&sz=728x90&wr=j&t=j&u=http%3A%2F%2Fwww.autoefax.com%2Fsearch%2Fnew%2Fbentley%2Findex.php&r=&rnd=761841 HTTP/1.0" 302 - "http://www.autoefax.com/search/new/bentley/index.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Alexa Toolbar; InfoPath.2)" 221.224.78.227 - - [04/May/2009:14:31:14 +0000] "GET http://afe.specificclick.net/test.php HTTP/1.0" 200 4125 "http://www.autoefax.com/search/new/gmc/envoy.php" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)" 221.224.78.227 - - [04/May/2009:14:31:14 +0000] "GET http://afe.specificclick.net/test.php HTTP/1.0" 200 4125 "http://www.autoefax.com/lease/rebates/make/pontiac.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1" 221.224.78.227 - - [04/May/2009:14:31:14 +0000] "GET http://afe.specificclick.net/test.php HTTP/1.0" 200 4125 "http://www.autoefax.com/lease/rebates/make/land rover.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12" 221.224.78.248 - - [04/May/2009:14:31:14 +0000] "GET http://afe.specificclick.net/test.php HTTP/1.0" 200 4125 "http://www.autoefax.com/search/new/aston martin/vantage.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1" 221.224.78.245 - - [04/May/2009:14:31:14 +0000] "GET http://afe.specificclick.net/login.php HTTP/1.0" 302 - "http://www.autoefax.com/search/new/bentley/index.php" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Alexa Toolbar; InfoPath.2)" 117.15.29.77 - - [04/May/2009:14:31:16 +0000] "GET http://clickboothlnk.com/img/wegive/1/1.jpg HTTP/1.0" 404 346 "http://newcareerworld.com" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/85 (KHTML, like Gecko) Safari/85"

Here is the proxy.conf (no changes but here it is for proof):

$ cat /etc/apache2/mods-enabled/proxy.conf <IfModule mod_proxy.c> #turning ProxyRequests on and allowing proxying from all may allow #spammers to use your proxy to send email.
    ProxyRequests Off

    &lt;Proxy *&gt;
            AddDefaultCharset off
            Order deny,allow
            Deny from all
    &lt;/Proxy&gt;

    # Enable/disable the handling of HTTP/1.1 &quot;Via:&quot; headers.
    # (&quot;Full&quot; adds the server version; &quot;Block&quot; removes all outgoing Via: headers)
    # Set to one of: Off | On | Full | Block

    ProxyVia On

</IfModule>

Here is the virtual host conf:

$ cat example01.org.conf <VirtualHost 192.168.3.56:80> SuexecUserGroup "#1001" "#1002" ServerName example01.org ServerAlias www.example01.org ServerAlias webmail.example01.org ServerAlias admin.example01.org ServerAlias lists.example01.org DocumentRoot /home/vhost1/domains/example01.org/public_html ErrorLog /var/log/virtualmin/example01.org_error_log CustomLog /var/log/virtualmin/example01.org_access_log combined ScriptAlias /cgi-bin/ /home/vhost1/domains/example01.org/cgi-bin/ ScriptAlias /awstats /home/vhost1/domains/example01.org/cgi-bin DirectoryIndex index.html index.htm index.php index.php4 index.php5 <Directory /home/vhost1/domains/example01.org/public_html> Options -Indexes IncludesNOEXEC FollowSymLinks allow from all AllowOverride All </Directory> <Directory /home/vhost1/domains/example01.org/cgi-bin> allow from all </Directory> RewriteEngine on RewriteCond %{HTTP_HOST} =webmail.example01.org RewriteRule ^(.*) https://example01.org:20000/ [R] RewriteCond %{HTTP_HOST} =admin.example01.org RewriteRule ^(.*) https://example01.org:10000/ [R] Alias /dav /home/vhost1/domains/example01.org/public_html Alias /pipermail /var/lib/mailman/archives/public <Location /dav> DAV On AuthType Basic AuthName example01.org AuthUserFile /home/vhost1/domains/example01.org/etc/dav.digest.passwd Require valid-user ForceType text/plain Satisfy All </Location> <Files awstats.pl> AuthName "example01.org statistics" AuthType Basic AuthUserFile /home/vhost1/domains/example01.org/.awstats-htpasswd require valid-user </Files> RedirectMatch /cgi-bin/mailman/([^/]*)(.*) https://example01.org:10000/virtualmin-mailman/unauthenticated/$1.cgi$2 RedirectMatch /mailman/([^/]*)(.*) https://example01.org:10000/virtualmin-mailman/unauthenticated/$1.cgi$2 ProxyPreserveHost On ProxyPass / http://example01/example01/ ProxyPassReverse / http://example01/example01/ <Proxy *> allow from all </Proxy> </VirtualHost>

I’m totally confused.

What exactly is the problem? (It sounds like you’re saying proxying is a problem…but you’ve turned on proxying to that internal server, and proxying through to it should be exactly what it does.)

Are you saying that your internal server is behaving as an open proxy on behalf of users?

Perhaps if you explained which of the hosts in that log file are yours and which are not (if my open proxy theory is actually what you’re asking about), we’d be able to understand a bit more about what is happening here. On first glance I assumed everything in the log was your servers…but on a third reading it sounds like you’re a bit too panicked for that to be the case. :wink:

Yes I think I am running an open proxy. Or possibly being/have been exploited. The proxying works as it should for my purposes, but it seems to have opened up to the world rather than just the internal network. This server is being hammered by multiple hosts with funny looking GET requests. I can’t say for sure if it is actually being used successfully, but what I can say is that my IP was definitely tagged as a possible open proxy by some evil nework. The log posted is a sample of the bad traffic that I am getting in the log. None of those have to do with me or my network. I’m not running any ad services.

Also, I tried testing it out the only way I know how. Enter my IP and port 80 into Firefox network proxy settings. It does not work. I can’t figure out how they are using it as a proxy (if they are at all)

And one more thing. Thank you for this wonderful, wonderful product.

I don’t see anything that would open up Apache for use as a proxy in that configuration. You might check to be sure you don’t have ProxyRequests set to on anywhere else in the Apache configuration:

grep -r ProxyRequests /etc/apache2/*

And, you might also turn off ProxyVia, as it is normally only used in Forward Proxy situations, and might be contributing to the problem somehow.

While we figure this out, if there are recurring themes to the sites being proxied to, you could add ProxyBlock rules for those specific sites.

http://httpd.apache.org/docs/2.0/mod/mod_proxy.html#proxyblock

There are no additional ProxyRequests other than the one in proxy.conf which is turned off, so that looks good

I have turned off ProxyVia and it has not stopped the requests.

This has been going on for a few days so I have a very good number of hosts to block, I’ll be doing that next as a band-aid until I get this figured out.

Here is another clip of the log. What do you make of all the 404s?

06.15.234.78 - - [05/May/2009:00:04:57 +0000] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0&section=289946&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 332 "http%3A%2F%2Fwww.vafq.org%2Findex.html" "Mozilla/4.76 [en] (X11; U; HP-UX B.10.20 9000/782)" 61.250.139.252 - - [05/May/2009:00:04:59 +0000] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0&section=289946&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 332 "http%3A%2F%2Fwww.vafq.org%2Findex.html" "Mozilla/4.75 [en] (Win98; U)" 123.53.36.75 - - [05/May/2009:00:04:59 +0000] "GET http://i.netpartner.com/it?z=31vw191137&ao=6w4nk6n0g3&cr=11272&s=SUBID HTTP/1.1" 404 262 "http://us.f656.mail.yahoo.com/ym/ShowLetter?MsgId=0178_538077_9989_5545_10932_6_11_401997_19146346&Idx=1&YY=31965&y5beta=yes&y5beta=yes&inc=25&order=down&sort=date&pos=0&view=a&head=b&box=Inbox" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" 216.64.195.212 - - [05/May/2009:00:05:00 +0000] "GET http://ad.reachjunction.com/st?ad_type=pop&ad_size=0x0&section=505085&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 333 "http%3A%2F%2Fwww.rsfox.com%2Findex.html" "Mozilla/4.76 [en] (Windows NT 5.0; U)" 206.15.234.78 - - [05/May/2009:00:05:02 +0000] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0&section=289946&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 332 "http%3A%2F%2Fwww.vafq.org%2Findex.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)" 216.64.195.212 - - [05/May/2009:00:05:06 +0000] "GET http://ad.reachjunction.com/st?ad_type=pop&ad_size=0x0&section=505085&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 333 "http%3A%2F%2Fwww.rsfox.com%2Findex.html" "Mozilla/4.76 [en] (X11; U; HP-UX B.10.20 9000/782)" 216.64.195.212 - - [05/May/2009:00:05:06 +0000] "GET http://ad.reachjunction.com/st?ad_type=pop&ad_size=0x0&section=11110&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 333 "index.php%2Fhttp%3A%2F%2Fwww.multidl.com" "Mozilla/4.0 (compatible; MSIE 5.5; AOL 6.0; Windows 98; Win 9x 4.90)" 216.64.195.212 - - [05/May/2009:00:05:08 +0000] "GET http://ad.reachjunction.com/st?ad_type=pop&ad_size=0x0&section=505085&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 333 "http%3A%2F%2Fwww.rsfox.com%2Findex.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 2.0.40" 216.64.195.212 - - [05/May/2009:00:05:09 +0000] "GET http://ad.reachjunction.com/st?ad_type=pop&ad_size=0x0&section=505085&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 333 "http%3A%2F%2Fwww.rsfox.com%2Findex.html" "Mozilla/4.76 [en] (X11; U; HP-UX B.10.20 9000/782)" 216.64.195.212 - - [05/May/2009:00:05:14 +0000] "GET http://ad.reachjunction.com/st?ad_type=pop&ad_size=0x0&section=505085&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 333 "http%3A%2F%2Fwww.rsfox.com%2Findex.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 2.0.40" 206.15.234.78 - - [05/May/2009:00:05:15 +0000] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0&section=289946&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 332 "http%3A%2F%2Fwww.vafq.org%2Findex.html" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)" 206.15.234.78 - - [05/May/2009:00:05:16 +0000] "GET http://ad.adserverplus.com/st?ad_type=pop&ad_size=0x0&section=289946&banned_pop_types=29&pop_times=1&pop_frequency=86400 HTTP/1.1" 404 332 "http%3A%2F%2Fwww.vafq.org%2Findex.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MyIE2; TencentT"

This forum doesn’t let you edit posts. I just wanted to say that the 404s have been happening before I made the changed to ProxyVia.

Also here are some interesting URLs:
124.226.234.15 - - [05/May/2009:00:05:42 +0000] "GET http://clickingagent.com/proxycheck.php?ip=my.ip.xxx.xxx&port=80&loc= HTTP/1.0" 404 342 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
221.0.90.194 - - [05/May/2009:00:05:50 +0000] "GET http://www.accessteams.com/proxyheader.php HTTP/1.0" 404 345 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
222.187.105.197 - - [05/May/2009:00:07:25 +0000] "GET http://clickingagent.com/proxycheck.php?ip=my.ip.xxx.xxx&port=80&loc= HTTP/1.0" 404 342 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
210.197.164.15 - - [05/May/2009:00:08:37 +0000] "GET http://www.cjpjp.com/proxyheader.php HTTP/1.0" 404 339 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

I replaced my IP with my.ip.xxx.xxx

Hey Sauce,
Did you ever find out what the underlying issue was? I have an almost identical issue (judging by the logfiles) that just cropped up right around the exact time yours did. The only variant is that one of the servers is running Tomcat (apache variant) and the other is running an IIS server with ColdFusion. These were vendor setup, but we did a google search on our domain name and +proxy, and we were all over the place. This is with us having the PIX setup to only allow port 80 and 443 traffic through the conduit. Both of these boxes are in the internal network, but have translations through the PIX to them. At one point, we had over 2800 active connections on one box and 2400 on the other. It was bringing our 45mb pipe to its knees.

Our logs look pretty much identical, and the only way we’ve managed to even throttle some of the traffic is to force everything through SSL, and block port 80 - which is not really a true fix. I apologize for posting on these forums, but I’ve yet to see much other information or suggestions elsewhere on the web - we’ve been combating this for a few weeks now. Any suggestions at this point would be greatly appreciated.

Thanks

This forum was down for a few days so I wasn’t able to post back.

Joe noticed my "ProxyVia" got turned on in proxy.conf. Default is off. I probably turned it on by accident. Once I turned it off, my problems were solved.

This forum was down for a few days so I wasn't able to post back.

We were never down, as far as I know. (I certainly managed to post a couple dozen messages per day.) :wink: