Kinesis & Streaming
15 questionsKinesis Data Streams, Firehose, and MSK for real-time ingestion on AWS.
Shard Capacity and Throughput Limits
A Kinesis Data Stream has 4 shards. What is the maximum write and read throughput? What happens if a producer exceeds the write limit on a single shard?
Partition Key Strategy
You're ingesting clickstream data from a website with 50M daily users. How would you choose a partition key for Kinesis Data Streams, and why does it matter?
Firehose Buffering and Transforms
Explain how Kinesis Data Firehose buffers data before delivery. How would you add a transformation (e.g., converting JSON to Parquet) in the pipeline?
Firehose vs Data Streams
When would you choose Kinesis Data Firehose over Kinesis Data Streams? Give specific use cases for each.
KCL Checkpointing
Explain how the Kinesis Client Library (KCL) manages checkpointing. What happens if a KCL worker crashes mid-processing? How do you handle duplicates?
Enhanced Fan-Out
What is Kinesis Enhanced Fan-Out? When would you use it vs. the shared throughput model? What are the cost implications?
MSK Serverless vs Provisioned
Compare MSK Serverless vs MSK Provisioned. When would you recommend each? What are the limitations of MSK Serverless?
MSK Connect for Data Pipelines
What is MSK Connect? How would you use it to build a pipeline from MSK to S3 without writing consumer code?
Kinesis vs Kafka Decision Framework
Your company needs a real-time streaming platform. The team is debating Kinesis Data Streams vs MSK (managed Kafka). What factors drive this decision?
Data Retention and Replay
What is the default and maximum data retention period for Kinesis Data Streams? How would you replay data from a stream?
Resharding Strategies
Your Kinesis stream has hot shards and you need to reshard. Explain the resharding process, its limitations, and how you would automate it.
Kinesis Data Analytics (Managed Flink)
What is Kinesis Data Analytics (now Managed Apache Flink)? How would you use it to compute real-time aggregations from a Kinesis stream?
Firehose to S3 with Partitioning
How would you configure Kinesis Data Firehose to deliver data to S3 with Hive-style partitioning by date? What file format and compression would you recommend?
Error Handling in Streaming Pipelines
Design an error handling strategy for a Kinesis-based streaming pipeline that ingests payment events. How do you handle poison pills, processing failures, and ensure no data loss?
Stream Replay and Reprocessing Architecture
Your team discovers a bug in a Kinesis consumer that has been silently dropping 5% of events for 3 days. Design a reprocessing strategy to recover the lost data.