A Few Noteworthy UNIX Commands

A few noteworthy UNIX commands I've used recently:

• parallel: Utilize your multiple {core,cpu}s!
• fdupes: "Finds duplicate files in a given set of directories"
• s3cmd sync: Like rsync, but with S3 as the file/dir destination

As it turns out you don't need much more than this to backup your photo archives to AWS S3.

I used parallel to speed up various batches of S3 operations whose per-request latency is noticeable for large batch sizes.

Note that s3cmd sync will not copy symbolic links to S3, but it will copy hard links, so fdupes -H can be used to eliminate uploading of these redundant files.

UNIX command to analyze Amazon S3 logs

I wanted to monitor the download activity on a particular file that I made publicly available on my Amazon S3 account.  Here's how I do it with from a bash command-line:

~/dev/s3-curl/s3curl.pl --id= -- https://s3.amazonaws.com/ 2> /dev/null | xpath -e '//Contents/Key/text()' 2> /dev/null | grep '^logs/' | xargs -i ~/dev/s3-curl/s3curl.pl --id= -- https://s3.amazonaws.com//{} 2> /dev/null | grep 'GET\.OBJECT.*'

where and are variables representing the name of your S3 bucket and the filename you'd like to monitor and is the name of the profile you configured in your ~/.s3curl file.

The above command assumes these S3 Logging options for the bucket:

• Enabled is checked
• Target Bucket is the same as the bucket containing the file being monitored
• Target Prefix is "logs/"

In a nutshell, this works by first downloading the file listing of the bucket, then extracting the log file names, then downloading each log file in turn and finally grep'ing them for the download activity (GET.OBJECT).

You'll need to have these command-line tools available:

• s3curl
• xpath