Backup to AWS S3 with s3cmd

Matt Foster — Sat, 14 Mar 2015 19:31:05 +0000

Particularly since the introduction of Glacier, S3 from Amazon is quite attractive as an offsite backup offering (archive the backups to Glacier automatically after, say, a week with lifecycle management and your storage costs drop dramatically).

Of course, we still have to keep an eye on our data transfer costs. There are two possible candidates for backing up our Linux Server/VPS to S3 that I’ve seen and used in the past, either: s3cmd or s3fs

S3FS certainly feels nice, and we can rsync to it in the normal way, but (and it is potentially a huge but – no pun intended) AWS S3 data charges are not just for storage, but also bandwidth transferred, and perhaps critically the number of requests made to the S3 API. I freely confess to having doing zero measurement on the subject, but it just feels instinctive that a FUSE filesystem implementation is going to make way more API calls than the python scripts that call the API directly that are s3cmd.

So using the rsync like logic you might consider doing something like:

cd /var/www/
s3cmd sync -r vhosts --delete-removed s3://$BUCKET/current/vhosts/

There is a small snag however to this approach. s3cmd keeps the directory structure in memory to help it with the rsync logic. This is fine if you are on real tin, with memory to spare. But on a VPS, especially an OpenVZ based one where there is no such thing as swap, this can be a real show stopper for large directory structures as the hundreds of MB of RAM required just are not available. Time for our old friend the OOM killer to rear it’s head ?

Recursion of some form would be the elegant answer here. However elegance is for those with time for it, and the following seems to work very effectively with minimal RAM consumption:

cd /var/www
for i in `find . -type d -links 2 | sort | sed -e 's/\.\///g'`
do
s3cmd sync -r $i/ --delete-removed s3://$BUCKET/current/vhosts/$i/
done

The find command looks for directories which only contain two directories (. and ..), that is to say they are the end nodes of a directory tree. And then we back them up, one by one.

Simples.

Other Service Providers are also Available

Matt Foster — Tue, 03 Mar 2015 20:00:24 +0000

Anyone who has worked with me in the past couple of years will know that I have a very strong preference for recommending Amazon AWS as your IaaS provider of choice. It is mature, robust, performant, and has a whole raft of PaaS type features to make things easy and lower the sysadmin burden/requirement.

It also represents really good value for money to my mind, and what better way to learn about it from the free usage tier (if you stay within the fairly generous limits it truly is free). Since the introduction of the t2.micro node, and general purpose SSD storage (replacing t1.micro, which was rather memory cramped, and our old friendly spinning rust) it is a serious piece of virtual hardware for a rather special price.

There is, however, no such thing as a one-size fits all answer. Perhaps you need a UK IP address. Perhaps you want a better pricing plan on TB of data in and out from your VPS. Perhaps you don’t need all the fancy infrastructure capabilities, but just want a few Linux boxen “in the cloud”. If so, you could do a lot worse than to look at linode.com. I first had a shell on a linode many many many years ago (it still works), and it seems to fit into the “it just works bucket”. Good price point (especially if data transfer is a worry for you), fast NIC speeds (getting over 100Mbps is challenging at this price level), ability to deploy images, a fabulous reporting/monitoring engine – Longview. And an API. Nobody should be touching anything that doesn’t have an API that you can do everything you need to through.

I do not, and have never, worked for either AWS or Linode, but they both have been wonderful providers to me and my clients time and time again.

VPS – Security Performance Architecture

Backup to AWS S3 with s3cmd

Other Service Providers are also Available