Large scale migration of encrypted objects in Amazon S3 using S3 Batch Operations
Many organizations have data governance strategies or compliance requirements that mandate their data be replicated and redundant across different management accounts and global regions. Moving encrypted data at scale can often take a few additional steps due to the need to decrypt and re-encrypt objects as part of the replication process.
Amazon Simple Storage Service (Amazon S3) offers three options for server-side encryption: Amazon S3 managed server-side encryption ( SSE-S3), server-side encryption with AWS KMS keys ( SSE-KMS), and server-side encryption with customer provided keys ( SSE-C). As of a recent update, SSE-S3 is applied automatically to all new objects as the default if you haven’t choosing another encryption method. You can easily perform large-scale Amazon S3 operations using Amazon S3 Batch Operations, including migrating or replicating your encrypted data to different accounts.
In this post, we walk through migrating new and existing S3 objects encrypted with SSE-KMS keys when the source and destination S3 buckets are owned by different AWS accounts in the same AWS Region. We accomplish this with S3 Batch Operations, which lets you perform large-scale batch operations on S3 objects. You can use the solution in this post to minimize latency by maintaining copies of your data in AWS Regions geographically closer to your users, to meet compliance and data sovereignty requirements, and to create additional disaster recovery resiliency.
Solution overview
Amazon S3 Batch Replication, through a Batch Operations job, provides a method for replicating objects that existed before a replication configuration was in place, objects that you have previously replicated, and objects that have failed replication. This solution helps you accomplish cross-account Amazon S3 Batch Replication.
- You will configure an Amazon S3 Replication rule that enables automatic, asynchronous copying of new encrypted S3 objects in your source S3 bucket in AWS account A to a destination S3 bucket in AWS account B.
- You will use Amazon S3 Batch Replication to replicate existing encrypted S3 objects in your source S3 bucket in AWS account A to a destination S3 bucket in AWS account B.
For further reading, check my blog @ https://aws.amazon.com/blogs/storage/large-scale-migration-of-encrypted-objects-in-amazon-s3-using-s3-batch-operations/
