This blog post will discuss how to copy files to Amazon's simple storage service (S3) using
Opscode Chef.  Awright, let's get krack-a-lackin!

This Blog Post Makes the Following Assumptions

1. You have successfully installed chef-client.
2. You have a working knife config.
3. You have either a working open source chef server or you're using enterprise chef.

Note: Enterprise Chef comes with 5 free nodes!

So let's start off by installing my amazon_s3cmd cookbook.

luke@alderaan:~ $ knife cookbook site install amazon_s3cmd

Next you'll need a secret key for your databag.

luke@alderaan:~ $ openssl rand -base64 512 > data_bag_secret_key

Now create a new data bag item that will be used.

skywalker@alderaan:~/your/chef-repo$ knife data bag create  --secret-file ~/data_bag_secret_key s3cmd s3cfg
Created data_bag[s3cmd]
Created data_bag_item[s3cfg]

If you get the following error below:

ERROR: RuntimeError: Please set EDITOR environment variable

Make sure you export your editor as EDITOR.

export EDITOR=vim

Verify your encrypted data bag items.

skywaler@alderaan:~/your/chef-repo$ knife data bag show s3cmd s3cfg
id:            s3cfg
s3_access_key:
  cipher:         aes-256-cbc
  encrypted_data:  BUNCH_OF_RANDOM_CHARS_HERE

iv: RANDOM_CHARS_HERE

version: 1 s3_secret_key: cipher: aes-256-cbc encrypted_data: BUNCH_OF_RANDOM_CHARS_HERE

iv: RANDOM_CHARS_HERE

version: 1 skywaler@alderaan:~/your/chef-repo$

Now check your decrypted data bag items.

skywaler@alderaan:~/your/chef-repo$ knife data bag show –secret-file=/home/you/data_bag_secret_key s3cmd s3cfg
id:            s3cfg
s3_access_key: YOUR_ACCESS_KEY_HERE
s3_secret_key: YOUR_SECRET_KEY_HERE

Copy your secret key to your node.

skywalker@alderaan:~ $ scp /home/you/data_bag_secret_key skywalker@alderaan:
skywalker@alderaan's password:
data_bag_secret_key

Move your key to /etc/chef.

skywalker@alderaan:~ $ sudo mv /home/skywalker/data_bag_secret_key /etc/chef/

Include the amazon_s3cmd::source recipe in your node's run_list if you want the latest beta version which supports adavnced features.

{
  "name":"my_node",
  "run_list": [
    "recipe[amazon_s3cmd::source]"
  ]
}

Run chef-client on your node to update it's configuration and install & configure s3cmd.

skywalker@alderaan:~ $ sudo chef-client

Confirm your s3cmd configuration

If you took the defaults, your s3cmd's config file will be located at /root/.s3cfg and should look something like the following.

skywalker@alderaan:~# sudo cat /root/.s3cfg
[default]
access_key = YOUR_ACCESS_KEY_HERE!
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
invalidate_on_cf = False
list_md5 = False
log_target_prefix =
mime_type =
multipart_chunk_size_mb = 15
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
recursive = False
recv_chunk = 4096
reduced_redundancy = False
secret_key = YOUR_SECRET_KEY_HERE!
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
urlencoding_mode = normal
use_https = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

BACK YO STUFF UP

If you made it this far; CONGRATS! You should now be ready to back files up to S3.

So, for example, let's say you have some backups in /mnt/backups you'd like to tar up and copy to S3.

skywalker@alderaan:~# ls /mnt/backups/
backup1.tar.gz  backup2.tar.gz

You can tar up your backups and confirm with the following commands.

skywalker@alderaan:~# tar zcvhf /tmp/backups.tar.gz /mnt/backups
tar: Removing leading `/' from member names
/mnt/backups/
/mnt/backups/backup1.tar.gz
/mnt/backups/backup2.tar.gz
jackl0phty:~# ls -alh /tmp/backups.tar.gz
-rw-r--r-- 1 root root 167 Dec 19 19:05 /tmp/backups.tar.gz

As you can see above, this will create a tar archive of the /mnt/backups directory and save it as /tmp/backups.tar.gz.

Next, let's create an S3 bucket that we can use to copy our backups to.

skywalker@alderaan:~# s3cmd mb s3://awesome-backups

Now, copy your backup to your S3 bucket.

skywalker@alderaan:~# s3cmd put /tmp/backups.tar.gz s3://awesome-backups

Note: Your bucket name awesome-backups must be globally unique in S3.

Finally, you should now be able to display the contents of your bucket.

skywalker@alderaan:~# s3cmd ls /tmp/backups.tar.gz s3://awesome-backups