Lambda & Step Functions
15 questionsLambda for event-driven ETL, Step Functions for serverless orchestration and workflow automation.
Lambda for S3 Event Processing
How would you use Lambda to process files as they land in S3? Walk through the architecture and key configuration settings.
Lambda Layers for Shared Dependencies
What are Lambda layers and how would you use them in a data engineering context? Give a practical example.
Cold Starts and Their Impact on ETL
What are Lambda cold starts? How do they impact data pipeline latency, and what can you do to mitigate them?
Memory and Timeout Tuning
You have a Lambda function that processes Parquet files from S3. It sometimes times out on larger files. How would you systematically tune memory and timeout settings?
Lambda + SQS Integration
How does Lambda process messages from an SQS queue? What happens if the Lambda function fails for some messages in a batch?
Step Functions for ETL Orchestration
Design a Step Functions state machine to orchestrate a daily ETL pipeline: ingest from S3, validate, transform with Glue, load to Redshift, and send notification. Include error handling.
Parallel and Map States
Explain the difference between Parallel and Map states in Step Functions. When would you use Distributed Map for a data pipeline?
Error Handling and Retry in Step Functions
How does Step Functions handle errors? Explain Retry, Catch, and how you'd implement a robust error handling strategy for a production data pipeline.
Express vs Standard Workflows
Compare Step Functions Express and Standard workflows. Which would you choose for a high-volume event-driven data pipeline?
Lambda vs Glue Decision
You need to transform data landing in S3. How do you decide between Lambda and Glue for the ETL job? What are the breakpoints?
Lambda Destinations and Async Invocation
Explain Lambda async invocation and destinations. How do they differ from DLQs, and when would you use them in a data pipeline?
Provisioned Concurrency
What is Lambda provisioned concurrency? When would you use it for data pipelines, and how do you calculate the right number?
Step Functions + Glue/EMR Integration
Design a Step Functions workflow that orchestrates a complex ETL: runs Glue crawlers, then parallel Glue jobs, then an EMR Spark job, with data quality gates between each stage.
Event Source Mappings Deep Dive
Compare Lambda event source mappings for SQS, Kinesis, and DynamoDB Streams. How do they differ in batching, error handling, and scaling?
Lambda Power Tuning
You have 50 Lambda functions across your data platform. How would you systematically optimize their memory/cost configurations? Describe the approach and tooling.