Monitoring & Security
12 questionsCloudWatch, IAM, KMS, and CloudTrail for securing and monitoring data pipelines.
CloudWatch Custom Metrics for Data Pipelines
How would you use CloudWatch custom metrics to monitor a data pipeline? What metrics would you track?
CloudWatch Logs Insights
How do you use CloudWatch Logs Insights to debug a data pipeline failure? Write a query to find error patterns.
Alarms for Data Freshness
How would you set up CloudWatch alarms to detect stale data in your pipeline? Include both the metric and alarm configuration.
IAM Roles for Cross-Service Data Access
A Glue job needs to read from S3, write to Redshift, and publish metrics to CloudWatch. How do you configure IAM for this?
Least Privilege for Data Pipelines
How do you implement least privilege in a data platform with dozens of pipelines? What tools and patterns help?
KMS: CMK vs AWS Managed Keys
Compare AWS managed keys with customer managed keys (CMK) in KMS. When would you use each for data at rest?
S3 Encryption Options
What are the S3 encryption options for data at rest? Which would you recommend for a data lake?
CloudTrail for Data Pipeline Audit
How do you use CloudTrail to audit data access in your pipeline? What events should you monitor?
GDPR and PII in Data Pipelines
How do you handle PII in a data pipeline to comply with GDPR? Describe technical controls from ingestion to analytics.
VPC Endpoints for Data Services
Why would you use VPC endpoints for data services like S3 and DynamoDB? How do they improve security?
CloudWatch Anomaly Detection for Pipelines
How does CloudWatch anomaly detection work? How would you use it instead of static thresholds for pipeline monitoring?
End-to-End Pipeline Security Architecture
Design a security architecture for a data pipeline that ingests sensitive financial data from an RDS database, processes it through Glue, stores in S3, and serves through Redshift. Cover encryption, access control, and audit.