Thanks a lot, I’m staying in touch with AWS Support so we can work on it. I have also enabled S3 logging to gather complete info and what I noticed is that lot’s of responses for PUT and DELETE are not 2xx but 301 Permament redirects. I’m not very familar with AWS API so I can’t tell if it is an issue or not but I think that 301 redirect are actually empty besides the headers so maybe that is at least a hint for you.
EDIT:
Below is a filtered log from S3 for a single domain that has failed last night to upload. I have cenzored private information. S3 logs to a set of files so I just did cat * > file on them and the log entries are not in order but since the time stamps are not very precise anyway I just did not bother to sort it as it does not seem to make sense anyway. What seems interesting to me is that all the entries with 301 redirects have the “requester” field empty (dash mark between IP and RequesID) which, I think, indicates an issue somewhere in the area of authentication against S3. S3 log format explanation I used: http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html .
Just a word to report that, for the first time since the issue appeared (end of May), no backup failed last night, after changing “S3 retry” from 3 to 10.
I received another response from AWS support this night. I mentioned them that I noticed the pattern that the logs indicate authentication issue as the redirects are issued to the requests without the “requester” field. AWS Support responded as follows:
"I can see from the logs that some requests are not authenticated (sent anonymously) and S3 responds with redirect for these requests.
Probably there are authentication headers missing, as I would expect different answer for wrong signature.
You are right, there are request IDs in the log.
Unfortunately these are useless to us without second part of the ID, which is twice as long and is returned in “x-amz-id-2” header [1].
Even then, I doubt if any valuable information is there for us, as we don’t log HTTP headers, so we will only see some “Unathenticated” status, similar to the log you provided."
I will pass along the above info from Amazon to Jamie.
Sorry for the delay in providing a way to view the headers – Jamie is on vacation now, and although he’s still working while away, his communication is a bit more delayed
He and I have been discussing how best to build what Amazon is requesting, and late last night Jamie responded with a patch. With the updated code included in that patch, anytime the above error is thrown, /var/webmin/miniserv.error will contain the headers sent to and from Amazon during the failed request.
This is the patch, and I’ll attach the full s3-lib.pl as an attachment to this post:
No changes on my side, we still observe the issue. Recently I raised the amount of repeated attempts to 15 which seems to help for incremental backups but the full weekly backups never make it.
Eric, is there anything else I can do to help you guys resolve this issue?
I have Debian 7 server with Virtualmin 4.10 GPL and have the same issue. Backups are randomly failing with random vservers. Even not so big incremental backups are failing from time to time.
The debugging output we’ve been seeing hasn’t led to any sort of resolution… it certainly appears that all requests to S3 are being authenticated, and that, in some cases, the response is blank. Which is very odd!
What Jamie is doing is revamping the S3 support to use the latest code available in the S3 library that it was based on. Virtualmin’s S3 support was built some years ago, and not much with it has changed since then.
So to rule out any sort of unusual compatibility issue, Jamie is redoing it to include all the changes and updates in the S3 library. Let’s see if that helps!
This thread isn’t too old to add to I hope, but I’ve been experiencing irregular backup issues since July I think. Always “no buffer space available” errors after 10-15 of 20 vhosts. A 2nd box with even more vhosts backs up smooth as butter.
Uploading archive to Amazon’s S3 service …
… upload failed! HTTP connection to s3.amazonaws.com:443 for /backup1/10-12-2014/domain.com.tar.gz failed : Failed to connect to s3.amazonaws.com:443 : No buffer space available
I’ve tried all sorts of experimenting with the backup settings - even splitting the entire schedule into two different nights. Nothing makes a difference.
My question is simply, does a timeline exist for the rollout of Jamie’s S£ updates as mentioned in the last post?
I am getting exactly the same issue as in this thread on occasion as well. I just recently replaced one of my backup servers with S3. It made sense in terms of price and reliability. Except now I have this issue.
Hi.
i had the same problem with D.O. kvm ubuntu 12.04 image. Virtualmin 4.11 latest.
found this thread so i have changed retry number to 10 but only solved 2 more vhosts. success was 11 of 20 always. so i kept examining the logs & i thought it’s about filesize of vhost backup and if it is bigger than chunk size that problem appears more. so i have changed chunk size to 100Mb. now i don’t get that error. several times had backup. all fine now. please try & confirm.
@monsieurQ: Just be sure your chunk size is larger than your vhost backup file size. ( I believe that is the problem, chunking the backup file size to upload makes problem on S3 side. i think it’s S3 bc i am also developer related with S3 & l respect JamieCameron more than Amazon )
Thanks monsieurQ and risyasin, Ive increased chunk size from default 5MB to 100MB. However my backup sizes are much larger than 5MB and fails happen only from time to time. Ill monitor the way things going for the next week and report any results.
@risyasin “l respect JamieCameron more than Amazon :)”
Well said
One of my vhosts is 500mb. Is there any reason I could/should not set chunk size to say 750mb for example? Would this have a negative effect on other vhosts?
@RealGecko Thanks - please do report back any progress
Having previously got the “No buffer space available” error with up to 10 vhosts failing, significantly upping the chunk size has resulted in only 2-3 vhosts failing on average with the “upload failed! Empty response to HTTP request” error.
I know this particular error is what most of you have been getting anyway but this is actually progress for me!
