S3 Backup Failing

[jamest]

SYSTEM INFORMATION
OS type and version:	Ubuntu 18.04.6 LTS
Webmin version:	1.981
Virtualmin version:	6.17.gpl-3

---

On just a single server we have an issue where S3 backups are failing to upload. Virtualmin only responds with:

Uploading archive to Amazon's S3 service ..

.. upload failed!
Invalid HTTP response : HTTP/1.1 400 Bad Request Backup failed!

See the progress output above for the reason why.

Triple checked credentials and have tried with S3 encryption on and off.

Using the aws cli tool on command line works fine, but fails through Virtualmin.

We enabled debug logging in Webmin and could see that there were multiple TCP connections to AWS but the logs didn’t give anything of use.

The backup file is just 4MB so filesize shouldn’t cause any issues.

Backup file:

file=/etc/webmin/virtual-server/backups/148880307532283
days=*
parent=1
enabled=2
strftime=1
months=*
key=
onebyone=1
feature_all=0
increment=1
mins=0
hours=4
plan=
kill=0
email=email@example.com
fmt=1
special=
mkdir=1
purge=
doms=
interval=
exclude=
email_err=1
all=1
email_doms=
after=
errors=0
weekdays=*
before=
id=148880307532283
virtualmin=
features=mysql
dest=s3://[redacted]@s3-bucket-name/Database/%Y-%m-%d.database
ownrestore=0
compression=
reseller=
desc=

[Ilia]

Hi,

If you don’t use strftime as destination directory, does it change anything?

If you enable logging on S3 side, what do you get logged at the moment of running a backup?

[jamest]

With strftime off get the same.

AWS CLI

REDACTED BUCKETNAME [18/Oct/2021:07:08:37 +0000] SERVERIP arn:aws:iam:REDACTED:user/REDACTED REDACTED REST.PUT.OBJECT PATH/TO/FILE "PUT /PATH/TO/FILE HTTP/1.1" 200 - - 1862 84 27 "-" "aws-cli/1.16.290 Python/3.6.9 Linux/4.15.0-159-generic botocore/1.13.26" - REDACTED SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader BUCKETNAME.s3.eu-west-2.amazonaws.com TLSv1.2 -

Virtualmin backup

REDACTED BUCKETNAME [18/Oct/2021:06:54:43 +0000] SERVERIP  - REDACTED REST.PUT.OBJECT PATH/TO/FILE "PUT /PATH/TO/FILE HTTP/1.0" 400 InvalidRequest 324 - 3 - "-" "-" - REDACTED= SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader BUCKETNAME.s3.eu-west-2.amazonaws.com TLSv1.2 -

Also have this logged followed by a couple of attempts at Virtualmin trying to delete the file off S3.

REDACTED BUCKETNAME [18/Oct/2021:06:53:13 +0000] SERVERIP  - REDACTED REST.PUT.OBJECT /PATH/TO/FILE "PUT /PATH/TO/FILE HTTP/1.0" 301 PermanentRedirect 461 - 2 - "-" "-" - REDACTED - ECDHE-RSA-AES128-GCM-SHA256 - s3.amazonaws.com TLSv1.2 -

Obvious differences to me are CLI uses SigV4 and passes in the arn:aws:iam:REDACTED:user/REDACTED

[jamest]

Managed to find the problem.

Turns out that at some point Virtualmin had saved the wrong secret key in /root/.aws/credentials despite what has been put into the settings on the backup form which I guess is only used by the backup Perl script but when it uses the awscli that’s ignored and it uses the .aws/credentials file instead

[unborn]

so it was not the virtualmin fail, basically you really read it wrong.

[jamest]

Read what wrong? All the virtualmin files had the correct credentials showing.

At some point when creating a backup to S3, virtualmin must save the credentials in /root/.aws/credentials. I assume I pasted the access key in both key+secret at first then after updating the secret key virtualmin updated its own files but not the /.aws/credentials file.

Virtualmin should probably be updating the .aws/credentials file if details are changed within virtualmin.

[unborn]

as well said by me and makred your solution, in end of the day mistake was done on your end… you did saved wrong key, remember API just do what you want. and I thinking virtualmin API is going like 10 years straight… without issues. that was my 2pi for you. end of the day you realise that issue and was able to fix it without anyone…so correctly saying it was you not virtualmin

[jamest]

Your response comes off as rude and isn’t warranted. You had nothing useful to add so you didn’t need to comment.

And with regards to it not being a virtualmin problem, it is a problem. Virtualmin isn’t updating credentials, only saving on initial setting up. It also isn’t pulling through the error from S3 that credentials are wrong but just shows a 400 bad request error.

A user making a small mistake on set up shouldn’t be irreversible. Every other issue I’ve had with Virtualmin has been solvable through UI.

[unborn]

@jamest

I’m not aware of anything to be offending towards you or others nor to be rude. if you feel offended or that I was rude to you please accept my apologies and I wish you could solve your issues here on forum. have pleasent day.

[martbean]

THANK YOU! This solved it for me. /root/.aws/credentials looked like this:

[MYACCESSKEY]
aws_access_key_id = MYACCESSKEY
aws_secret_access_key = MYACCESSKEY
[default]
aws_access_key_id = MYACCESSKEY
aws_secret_access_key = MYSECRETACCESSKEY

I copied the real secret access key into the first section and backing up to S3 now works. I don’t recall getting the secret access key wrong when I set it up (though it was a while ago) but it was certainly correct in Virtualmin.

I find it strange that I was getting an error message telling me to use AWS4-HMAC-SHA256 as opposed to an error that my secret key was wrong but at least it’s working.