Sunday, November 3, 2013

A Few Noteworthy UNIX Commands

A few noteworthy UNIX commands I've used recently:

  • parallel: Utilize your multiple {core,cpu}s!
  • fdupes: "Finds duplicate files in a given set of directories"
  • s3cmd sync: Like rsync, but with S3 as the file/dir destination
As it turns out you don't need much more than this to backup your photo archives to AWS S3.

I used parallel to speed up various batches of S3 operations whose per-request latency is noticeable for large batch sizes.

Note that s3cmd sync will not copy symbolic links to S3, but it will copy hard links, so fdupes -H can be used to eliminate uploading of these redundant files.

No comments: