Manage Amazon S3 storage costs granularly and at scale using S3 Intelligent-Tiering
Cost-effective data storage is critical when building and scaling data lakes that manage and hold growing datasets. By choosing the right storage architecture, customers are empowered to quickly experiment and migrate to AWS. Amazon S3 Intelligent-Tiering is a storage class that allows customers to optimize storage costs automatically when data access patterns change without performance impact or operational overhead, for all stages of data lake workflows.
In this blog, we explain how developers and cloud operations managers can use S3 Intelligent-Tiering to optimize storage costs. We start off by breaking down S3 Intelligent-Tiering access tiers. We then focus on multiple use cases, starting with individual buckets and directly uploading objects to S3 Intelligent-Tiering. Following that, we also explain how to transition existing objects from S3 Standard or S3 Standard-IA to S3 Intelligent-Tiering, using an S3 Lifecycle policy.
Later on we explain how to enable S3 Intelligent-Tiering Lifecycle policy at scale, on a large number of buckets. Here we cover two scenarios, namely transitioning objects between S3 Intelligent-Tiering access tiers based on access patterns, for both existing and new buckets. These use cases enable developers and cloud operations managers to manage the S3 Intelligent-Tiering storage class configurations on individual S3 buckets or at scale across multiple S3 buckets in an AWS account, optimizing storage costs automatically when data access patterns change.
S3 Intelligent-Tiering access tiers
S3 Intelligent-Tiering automatically stores objects in three access tiers:
- Frequent Access tier optimized for frequently accessed data
- Lower-cost Infrequent Access tier optimized for infrequently accessed data
- Very-low-cost Archive Instant Access tier optimized for rarely accessed data
To save more on storage cost that doesn’t require immediate retrieval, you can activate the optional asynchronous Archive Access and Deep Archive Access tiers. When turned on, objects not accessed for 90 days are moved directly to the Archive Access Tier (bypassing the automatic Archive Instant Access tier) and the Deep Archive Access tier after 180 days.
There are no data retrieval charges in S3 Intelligent-Tiering. Customers can implement S3 Intelligent-Tiering with a small monthly per-object fee for monitoring and automation and has a minimum eligible object size of 128 KB for auto-tiering. Amazon S3 Intelligent-Tiering further optimizes storage cost savings by removing the minimum storage duration and monitoring and automation charge for objects less than 128 KB.
- Frequent Access tier (automatic): This is the default access tier that any object created or transitioned to S3 Intelligent-Tiering begins its lifecycle in.
- Infrequent Access tier (automatic): If an object is not accessed for 30 consecutive days, the object moves to the Infrequent Access tier.
- Archive Instant Access tier (automatic): If an object is not accessed for 90 consecutive days, the object moves to the Archive Instant Access tier.
- Archive Access tier (optional): You can activate the Archive Access tier for data that can be accessed asynchronously. After activation, the Archive Access tier automatically archives objects that have not been accessed for a minimum of 90 consecutive days. You can extend the last access time for archiving to a maximum of 730 days. Standard retrieval times for this access tier can range from 3–5 hours. Expedited retrieval is an option if you need a faster access to the object.
- Deep Archive Access tier (optional): You can activate the Deep Archive Access tier for data that can be accessed asynchronously. After activation, the Deep Archive Access tier automatically archives objects that have not been accessed for a minimum of 180 consecutive days. You can extend the last access time for archiving to a maximum of 730 days. Standard retrieval of objects in this access tier occurs within 12 hours. Expedited retrieval is an option if you need a faster access to the object.
Solution overview
For further reading, check my blog @
