I’ve been having this for a few days:
Uploading archive to Amazon's S3 service ..
.. upload failed! Empty response to HTTP request
Under a single backup plan, all using the same credentials, backing up to the same bucket, one night vhost A fails, the other night vhost D, the other night F, then A again, etc.
What should I look into? The firewall doesn’t show anything (as it shouldn’t).
FWIW, I’ve also been having this intermittently for some months. Like your experience, there is no obvious pattern. Two different servers, each with a 20-30 virtual servers. Maybe a 1% failure to upload. Latest Virtualmin release 4.08 at present).
If anyone can suggest some diagnostic steps I’d be happy to try.
The only thing I know that changed recently was the following:
I went in via the mobile interface (on Android) to update the backup plans - I wanted to exclude a certain site from backup. When I saved the backup plan, every piece of information in the credential area got doubled up, like this: http://i.imgur.com/avc6XnY.png
Naturally, ALL backups failed the following day. So I went in and corrected this. They started working again. A few days later I started having this problem. Since the credentials have been corrected, I imagine that this past issue has nothing to do with the current one, but it’s nonetheless worth mentioning, even if just to bring attention to the mobile interface bug.
I have exactly the same problem.
The backups started failing on about June 11th.
Sometimes all backups are completed (I backup around 15 vhosts) but most of the time, at least one or two fail when trying to upload with the message already stated above:
Uploading archive to Amazon’s S3 service …
… upload failed! Empty response to HTTP request
Getting worse here :(. Failure rate per site backed up is slowly increasing.
martin
Aha! Mine started failing on June 14th. I can rule out the mobile interface bug, as that happened one month earlier (finally pulled logs from storage to check on this)
we need a fix 
thanks virtualmin team for all your hard work. i’m looking to convert to a paid version this year.
Backups to S3 were working, but we tried to create a new backup job on a new web server and now get - “upload failed! Empty response to HTTP request.” at the end as the uploaad begins. Trying to create a similar job on other web servers and we find they are all failing too with the same message.
Please keep us updated.
Webmin version 1.690 Virtualmin version 4.08.gpl GPL
Jamie tells me that this can happen if S3 closes the connection before returning a response
to a request.
One work-around is to increase the number of re-tries for S3
uploads, at System Settings -> Virtualmin Configuration -> Backup and Restore
-> Number of times to re-try FTP or S3 uploads.
I’m curious if increasing the re-tries resolves the issues you’re seeing.
-Eric
Increased from 3 to 6. will keep you posted, Eric. gracias.
Increasing from 3 to 6 seems to help for several VPS. Only one virtual server failed last night. Yesterday 11 failed.
The problem occurs on CentOS 5 VPS only, I have one CentOS 6 VPS that never had a problem with backups.
I have also increased from 3 to 6, has greatly reduced (but not eliminated) the symptoms. Running on CentOS 6.5, two different VPS.
Increase from 3 to 6 also helped, but did not solve the problem. Gerard, I’m also on CentOS 6.5
Same here, it helped but did not solve the problem. Changing from 6 to 10 doesn’t make a difference.
Now I guess it’s just wait for a fix from the Virtualmin team. Because the likelihood that the problem is on the S3 side are realistically negligible, given how long the problem has been occurring.
Howdy,
Jamie has been looking into this issue, though thus far he hasn’t found any awesome fixes that will make that issue go away.
Here’s the one thing he did come up with though – right now, the time between retries is 10 seconds.
We’re wondering if increasing that might help a bit.
So the next Virtualmin version will wait longer in between retries.
If you’d like to try that out now, the change to the Virtualmin code is very simple to implement that.
What you can do is edit the file s3-lib.pl that comes in the root Virtualmin directory, and on line 51 you’ll see this code:
sleep(10);
Try increasing that “10”… perhaps making that “30” would be a good place to start. And then be sure to let us know if that helps.
-Eric
I did that and I will report on it.
Thanks Eric.
I edited /usr/libexec/webmin/virtual-server/s3-lib.pl and increased the value to “30” like you suggested but it does’ seem to make a difference.
Should I have restarted Virtualmin / Webmin?
I’m still seeing issues after increasing to 30.
thanks
I also changed the retries from 3 to 6 on one of my servers, and a few of the virtual servers on it still failed backup
Howdy,
Thanks for your input!
I’ll talk to Jamie, but we’re unfortunately running low on ideas :-/
We haven’t been able to reproduce this issue in our testing, and there doesn’t seem to be any pattern to why it doesn’t work… especially since the same files might work the next day.
We understand that it’s not a fluke though, as it seems to semi-regularly happen to some people.
If any of you have some time to tinker (or perhaps are just sick of this not working :-), I’d be most curious to hear if anyone has better luck using the Rackspace Cloud backup option.
But we’ll continue to look into the S3 backup issues.
-Eric
