Amazon S3 compatible service

Hello,

I’m trying to setup Scaleway S3 as an Amazon replacement … I’ve read pretty much all topics on the subject and from what I understand, and because the Virtualmin GPL provides just for 2 S3 providers (Amazon and Rackspace), we have to tweak the “Amazon S3” cloud provider and give an alternate url in the “Edit Cloud Provider” screen (you have to click on the “Provider name” not the url …)

The field to edit is " S3-compatible server hostname" with a radio button to enable a text field where (my guess) I have to provide the Scaleway S3 url.

From what I read at Scaleway website, that url shoud be “https://s3.fr-par.scw.cloud” but if I try this I get an error: S3 compatible server must be a valid hostname

That url is definitely valid but when entered in a browser just displays a “The resource could not be found.” I’m not sure if that url should really go on a real website that can be displayed in a browser but the error given by Virtualmin seems to indicate that the url is not valid at all, but if I go to “scw.cloud” that does work perfectly with a real website … it’s like virtualmin does not recognize the “.cloud” top level domain … I even tried entering just “scw.cloud” and Virtualmin gives the same error.

Could it be that “.cloud” is not recognized as a real TLD by Virtualmin ?

Scaleway doc mentionning endpoint: Scaleway Object Storage - Scaleway

**Operating system:**Debian 10
Virtualmin 6.16

The field wants a hostname and not a URL.

Try changing this:

https://s3.fr-par.scw.cloud

To this:

s3.fr-par.scw.cloud

Oh … stupid of me !!!
I was mislead by the fact that the one for Amazon displays an url but yes you are perfectly right, it works when entered as a hostname !! My first test went through flawlessly after that :slight_smile:
Thank you very much !
Pierre.

Hello,

Well I spoke too fast, I was able to backup a small virtualhost no problems (a few megaoctets to test) but when trying something quite bigger (30go) it fails (twice):

Uploading archive to Amazon’s S3 service …
… upload failed! Part 1001 failed at 5242880000 : Upload failed : HTTP/1.1 400 Bad Request Try installing awscli package using package manager

Of course I installed “awscli” for my second attempt, installation went through ok but I’m wondering: how do I tell Virtualmin to use “awscli” ? is it automatic ?

Maybe I should have opened a new thread …

Pierre.

You usually just need to install a package and it should be accessible by any tool or package that depends on it in the system but there are packages that would need to be configured after installation too (e.g. the fail2ban package needs to be told what to check, where to check,how to respond, etc). I’m not too familiar with that particular package though but if the backup was successful after installing the package, then I think you’re most probably set.

Hello

Well the tar was created but upload to Scaleway failed at 5gb approx, before and after installing awscli.

I found in “System settings → Virtualmin configuration → Backup and restore” a field named “Full path to aws command for S3 operations” which just says “aws” so I guess that yes it’s installed and functioning.

At the same time I am asking Scaleway about a possible upload limit pertaining to the fact that I just created an account …

Pierre.

Hello,

Ok I got the solution ! In fact the interesting information in the error message was “Part 1001 failed”. Scaleway limits the size of each object to 5to (mine was 18gb in fact so no problem) but also limits the number of parts at 1000 … the default size of a part (a chunk) in Virtualmin is 5mb resulting for my example in 3600 parts … I modified the chunk size from 5mb to 1000mb (1gb) and it went through like a breeze with 18 parts :slight_smile:

Thanks for your help !
PC

1 Like

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.