S3
15 questionsPartitioning strategies, file formats, lifecycle policies, encryption, and access control for data lakes.
Hive-Style Partition Strategy
You're storing clickstream data in S3 for downstream analytics. The data arrives continuously and analysts query by date range. How would you partition this data in S3?
S3 Prefix Design for High Throughput
Your pipeline writes 50,000 small objects per second to S3. You're seeing throttling (HTTP 503 SlowDown). How do you fix this?
Small Files Problem in S3
Your Spark job writes data to S3 and creates 10,000 files of 1 MB each per partition. Downstream Athena queries are extremely slow. What's going on and how do you fix it?
S3 Select for Cost Reduction
You have a 10 GB CSV file in S3 and only need 3 columns from it. Currently you download the full file and filter in your application. How can you reduce data transfer and cost?
Lifecycle Policies and Storage Classes
Your data lake has 500 TB of data. Only the last 90 days are queried frequently. Older data is accessed a few times per year for compliance. How do you optimize storage costs?
S3 Event Notifications for Pipeline Triggers
New data files land in S3 every few minutes. You need to trigger a Lambda function to process each file as it arrives. How do you set this up?
S3 Access Points
Your data lake bucket is shared across 15 teams. The bucket policy has grown to 18 KB and is becoming unmanageable. How would you simplify access management?
Bucket Policies vs IAM Policies
When should you use an S3 bucket policy versus an IAM policy to control access? Can you give examples of each?
S3 Encryption Options
Your compliance team requires all data at rest in S3 to be encrypted with customer-managed keys and full audit trails of key usage. Which encryption option do you choose and why?
S3 Inventory for Data Lake Auditing
You manage a data lake with 50 million objects. You need a weekly report of all objects, their sizes, storage classes, and encryption status. How do you approach this efficiently?
S3 Batch Operations
You need to copy 100 million objects from one S3 bucket to another, change their storage class, and add a new tag. How would you do this efficiently?
S3 Cross-Region Replication
Your company requires all production data to be replicated to a second region for disaster recovery. How do you set this up with S3, and what are the caveats?
S3 Versioning for Data Protection
A developer accidentally overwrote a critical dataset in S3. How could this have been prevented, and how do you recover the data?
Choosing File Formats for S3 Data Lake
You're designing a new data lake on S3. The data will be queried by Athena and Spark. What file format would you choose and why?
S3 Data Lake Architecture at Scale
You're architecting a data lake on S3 for a company processing 10 TB/day from 50 data sources. Walk me through your bucket strategy, naming conventions, access control, and lifecycle management.