Copy Contents of one S3 Bucket to Another.

Need to automate copying files from one Amazon S3 bucket to another? So did I. Everything I found on google, like this, was useless. Most of the scripts I found required downloading the objects first to the local machine and then reuploading them to the destination bucket. Unacceptable, especially if you are dealing with a large and or many files.

I’ve never written a line of Ruby before, but it seems like there are some great AWS libraries for it, so I decided to give it a shot. There is a cool library out there called right_aws. You can install it using “#gem install right_aws”. Then simply copy this script:

#!/usr/bin/env ruby
require 'right_aws'

        S3ID = "Your AWS ID Here"
        S3KEY = "Your AWS secret key"
        SRCBUCKET = "Source Bucket"
        DESTBUCKET = "Destination Bucket"

        s3 = RightAws::S3Interface.new(S3ID, S3KEY)
        objects = s3.list_bucket(SRCBUCKET)
        objects.each do |o|
        puts("Copying " +  o[:key])
        s3.copy(SRCBUCKET, o[:key], DESTBUCKET, o[:key])
        end
        puts("Done.")

Make sure the file is executable and you should be able to run it via command line on any unix system. To make a generic ruby script get rid of the first line.

I know its pretty brutish, probably sucks in more ways than one, but for now it works. And I think I like Ruby :D

Posted in: Linux, Programming, Technology

Tags: , , , , , , ,

This entry was posted on Wednesday, February 16th, 2011 at 10:11 pm and is filed under Linux, Programming, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

5 Responses to “Copy Contents of one S3 Bucket to Another.”

  1. Rohan Dey says:
    August 25, 2011 at 10:22 am

    Thanks for the quick script it works like a charm. All other tried tools are like junk.

    Reply
  2. Robert Bousquet says:
    September 11, 2011 at 1:05 pm

    Awesome. This works really well. Hooked it up to “whenever” schedule so it just syncs production assets to staging bucket every day.

    Reply
  3. Geo says:
    January 10, 2012 at 9:03 am

    Great script man! Very usefull ;)

    Reply
  4. Tim Olsen says:
    February 1, 2012 at 6:24 pm

    Your script copied only the first 1,000 items in my bucket (I have about a 100,000 items). Here is a modified version of your script which copies everything:

    require ‘right_aws’

    S3ID = “Your AWS ID Here”
    S3KEY = “Your AWS secret key”
    SRCBUCKET = “Source Bucket”
    DESTBUCKET = “Destination Bucket”

    s3 = RightAws::S3Interface.new(S3ID, S3KEY)
    s3.incrementally_list_bucket(SRCBUCKET) do |h|

    h[:contents].each do |o|
    puts(“Copying ” + o[:key])
    s3.copy(SRCBUCKET, o[:key], DESTBUCKET, o[:key])
    end
    end
    puts(“Done.”)

    Reply

Leave a Reply