A Few Noteworthy UNIX Commands

A few noteworthy UNIX commands I've used recently:

• parallel: Utilize your multiple {core,cpu}s!
• fdupes: "Finds duplicate files in a given set of directories"
• s3cmd sync: Like rsync, but with S3 as the file/dir destination

As it turns out you don't need much more than this to backup your photo archives to AWS S3.

I used parallel to speed up various batches of S3 operations whose per-request latency is noticeable for large batch sizes.

Note that s3cmd sync will not copy symbolic links to S3, but it will copy hard links, so fdupes -H can be used to eliminate uploading of these redundant files.