Problems with AWS S3 even with awscli installed

Virtualmin
1
SYSTEM INFORMATION
OS type and version Ubuntu Linux 20.04.3
Webmin version 1.990
Virtualmin version 6.17-3

Hi,

I originally had problems with the London S3 and got … “Invalid HTTP response : HTTP/1.1 400 Bad Request” then I switched to the Ireland S3 which was more successful, ie. some of my subdomains did get uploaded, but others gave “Empty response to HTTP request. Headers were :” (it’s says “headers were” but they’re never shown). At least some subdomains were backing up without errors but I wanted a full backup so I tried switching to the USA S3 servers (east coast) only to get the original London error again with nothing backing up.

I have the latest AWS cli installed from official sources following official guide from Amazon. I have told Virtualmin where the awscli command is in its setup module.

The most recent try with the USA East coast S3 …

Backup failed! See the progress output above for the reason why. Total backup time was 2 hours, 25:24 minutes.
 
Virtual servers that failed :
    battlefield-classic.michaelzfreeman.org
    companyofheroes.michaelzfreeman.org
    dream.michaelzfreeman.org
    files.michaelzfreeman.org
    forgottenhope.michaelzfreeman.org
    freedom-eagle.michaelzfreeman.org
    fromthearchives.michaelzfreeman.org
    michaelzfreeman.org
    nextcloud.michaelzfreeman.org
    temp.michaelzfreeman.org
    ucas.michaelzfreeman.org
Sent by Virtualmin at: https://server.michaelzfreeman.org:10000
 
Creating backup for virtual server battlefield-classic.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database battlefield_classic ..
    .. done
 
    Dumping MySQL database battlefield_classic_wordpress ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 13 minutes, 25 seconds
 
Creating backup for virtual server companyofheroes.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 9 minutes, 27 seconds
 
Creating backup for virtual server dream.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 13 minutes, 16 seconds
 
Creating backup for virtual server files.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Empty response to HTTP request. Headers were :
 
.. completed in 17 minutes, 43 seconds
 
Creating backup for virtual server forgottenhope.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 9 minutes, 26 seconds
 
Creating backup for virtual server freedom-eagle.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database freedom_eagle ..
    .. done
 
    Dumping MySQL database freedom_eagle_wordpress ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 13 minutes, 35 seconds
 
Creating backup for virtual server fromthearchives.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 9 minutes, 25 seconds
 
Creating backup for virtual server michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Backing up Cron jobs ..
    .. none defined.
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database michaelzfreeman_org ..
    .. done
 
    Dumping MySQL database michaelzfreeman_org_wordpress ..
    .. done
 
    Copying Procmail and SpamAssassin configuration files ..
    .. done
 
    Backing up Webmin ACL files ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 17 minutes, 6 seconds
 
Creating backup for virtual server nextcloud.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database nextcloud ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Empty response to HTTP request. Headers were :
 
.. completed in 18 minutes, 39 seconds
 
Creating backup for virtual server temp.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Invalid HTTP response : HTTP/1.1 400 Bad Request
 
.. completed in 9 minutes, 26 seconds
 
Creating backup for virtual server ucas.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Empty response to HTTP request. Headers were :
 
.. completed in 13 minutes, 39 seconds
 
Deleting backups from %Y-%m-%d in Amazon S3 bucket michaelzfreeman-org-backup older than 3 days ..
    .. failed to list S3 files : Failed to list S3 files : The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

The try before that with Ireland …

Backup failed! See the progress output above for the reason why. Total backup time was 52 minutes, 39 seconds.
 
Virtual servers that failed :
    files.michaelzfreeman.org
    nextcloud.michaelzfreeman.org
Sent by Virtualmin at: https://server.michaelzfreeman.org:10000
 
Creating backup for virtual server battlefield-classic.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database battlefield_classic ..
    .. done
 
    Dumping MySQL database battlefield_classic_wordpress ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 42 seconds
 
Creating backup for virtual server companyofheroes.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 5 seconds
 
Creating backup for virtual server dream.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 26 seconds
 
Creating backup for virtual server files.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Empty response to HTTP request. Headers were :
 
.. completed in 16 minutes, 59 seconds
 
Creating backup for virtual server forgottenhope.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 5 seconds
 
Creating backup for virtual server freedom-eagle.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database freedom_eagle ..
    .. done
 
    Dumping MySQL database freedom_eagle_wordpress ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 53 seconds
 
Creating backup for virtual server fromthearchives.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 5 seconds
 
Creating backup for virtual server michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Backing up Cron jobs ..
    .. none defined.
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Webalizer configuration files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database michaelzfreeman_org ..
    .. done
 
    Dumping MySQL database michaelzfreeman_org_wordpress ..
    .. done
 
    Copying Procmail and SpamAssassin configuration files ..
    .. done
 
    Backing up Webmin ACL files ..
    .. done
 
    Backing up AWstats configuration file ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 14 minutes, 1 seconds
 
Creating backup for virtual server nextcloud.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying SSL Apache virtual host configuration and certificate ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Dumping MySQL database nextcloud ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. upload failed! Empty response to HTTP request. Headers were :
 
.. completed in 17 minutes, 39 seconds
 
Creating backup for virtual server temp.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 5 seconds
 
Creating backup for virtual server ucas.michaelzfreeman.org ..
    Copying virtual server configuration ..
    .. done
 
    Copying records in DNS domain ..
    .. done
 
    Saving mail aliases ..
    .. done
 
    Saving mail and FTP users ..
    .. done
 
    Backing up mail and FTP user Cron jobs ..
    .. none to backup
 
    Copying Apache virtual host configuration ..
    .. done
 
    Copying Apache log files ..
    .. done
 
    Copying Logrotate configuration ..
    .. done
 
    Creating TAR file of home directory ..
    .. done
 
    Uploading archive to Amazon's S3 service ..
    .. done
 
.. completed in 1 minutes, 22 seconds
 
Deleting backups from %Y-%m-%d in Amazon S3 bucket michaelzfreeman-dot-org-backup older than 3 days ..
.. found 27 backups, but none were old enough

Is a fix for this in the pipeline ? In the meantime is there any known workaround ? I already know about the different S3 servers workaround which is why I’m using Ireland.

1 Like
2

Hi,

Can you list buckets using CLI command, like aws s3api list-buckets command?

Does /root/.aws/credentials file contains correct credentials?

3

Thanks, no not until I’d done “aws configure” then I get …

 "Buckets": [
        {
            "Name": "michaelzfreeman-dot-org-backup",
            "CreationDate": "2022-03-14T08:29:19+00:00"
        },
        {
            "Name": "other-michaelzfreeman-backup",
            "CreationDate": "2022-03-07T19:20:56+00:00"
        }

Yet some backups have been successful even with this not set although I suppose Virtualmin could have been falling back to its built in S3 implementation.

4

There’s a bug in the current Virtualmin release that can cause this issue … but it will be fixed in version 7.0.

5

Thanks Jamie. That’s great to hear ! When is 7.0 expected to be rolled out ?

Cheers.

6

No exact date yet, sorry. Although, it’s happening soon. :wink:

7

OK, will look forward to it.

Is there any way I can patch the relevant module in Virtualmin myself ?

8

Yes, have a look at this patch:

9

Thanks Ilia ! :smile:

I just edited the perl file accordingly. I assume it will just get overwritten by an update normally and Virtualmin does not do any hash checking of the file ?

I also had an S3 credentials file in /root/.aws when the actual file that is updated by “aws configure” is in /home/nonrootuser/.aws. The one in root had both keys, access and secret, set to the same values. I assume Virtualmin creates the root aws files.

10

Sorry, It appears that I have completely misunderstood the initial problem!

Got it now. The simplest solution for your issue would be updating s3-lib.pl file completely and then restarting Webmin by using the following commands:

cd /usr/share/webmin/virtual-server
wget -N https://raw.githubusercontent.com/virtualmin/virtualmin-gpl/master/s3-lib.pl
/etc/webmin/restart
11

OK did that. Yes of course there are probably other updates in the perl file, not just the s3 variable, that I needed as well.

12

Yes of course there are probably other updates in the perl file, not just the s3 variable, that I needed as well.

No, that should be it for this particular bug. Does it fix your issue?

13

It does. My full site backup completed last night. No “backup failed” email this time ! Thanks for the help.

1 Like
14

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.