Backing up to S3 shows as completed and no errors – but no files in the bucket

I’ve been messing with the S3 backup functionality for a while now. I randomly noticed that the backups had stopped working properly after I started getting email messages about deleting old backups not working properly:

[snip] Creating TAR file of home directory .. .. done

Uploading archive to Amazon’s S3 service …
… done

… completed in 9 seconds

20 servers backed up successfully, 0 had errors.

Deleting backups from backups/servers/example.com/daily/%d-%b-%Y_%H-%M in Amazon S3 bucket example_bucket older than 7 days …
… failed to list S3 files :

But hey, the files had been uploaded, it just couldn’t delete the old backups, right?

After that had been occurring for a few days, I decided to look into the problem since I was tired of receiving emails every day. Then I noticed no new backups had been created since March 31st. So 7 days of backups had never taken place!

I then tried to delete the scheduled backup as well as the “Storage Provider”, and configuring everything again. At the same time I gave the IAM user maximum permissions for anything S3 related:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor1", "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::*", "arn:aws:s3:::*/*" ] } ] }

After running the backup, everything looks like it’s running smoothly, but the actual files aren’t created on S3. No files anywhere, nothing. No errors in the log output, which happily reports: “Uploading archive to Amazon’s S3 service … done”

Questions:

  • How can I enable any sort of debug logging or extended logging to debug this problem?
  • What are some steps I have missed that can debug this problem?
  • How can I get my backups to work again?

I tried to upgrade botocore which VirtualMin apparently uses, and I get this error:

# pip install --upgrade botocore Cleaning up... Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 290, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1097, in prepare_files req_to_install, self.upgrade) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement page = self._get_page(main_index_url, req) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page session=self.session, File "/usr/lib/python2.7/dist-packages/pip/index.py", line 694, in get_page req, link, "connection error: %s" % exc, url, TypeError: __str__ returned non-string (type Error)

Storing debug log for failure in /root/.pip/pip.log

That lead me on another journey, and as it turns out, this was caused by a borked /etc/ssl directory; after running update-ca-certificates --fresh, things started working again, including pip and s3 backups.