EMR
15 questionsCluster sizing, Spark tuning, Spot strategies, EMR on EKS, EMR Serverless, and monitoring.
Master, Core, and Task Node Roles
Explain the difference between master, core, and task nodes in an EMR cluster. How many of each would you configure for a production Spark workload?
Instance Fleet vs Instance Group
You're launching an EMR cluster and need to optimize for cost while maintaining availability. Should you use Instance Groups or Instance Fleets?
Spot Instance Strategy for EMR
Your nightly Spark batch job on EMR costs $500/night on On-Demand instances. How would you reduce costs by 60-70% using Spot instances without risking job failures?
EMR on EKS
Your organization is standardizing on Kubernetes. How does EMR on EKS compare to traditional EMR on EC2, and when would you choose it?
EMR Serverless
Your team wants to run Spark and Hive jobs without managing any cluster infrastructure. What is EMR Serverless and what are its trade-offs?
EMR Auto-Scaling Strategies
Your EMR cluster runs batch jobs with varying data volumes — some days 100 GB, others 2 TB. How do you configure auto-scaling to handle this efficiently?
Bootstrap Actions for Environment Setup
Your Spark jobs require specific Python libraries and system configurations that aren't in the default EMR AMI. How do you customize the environment?
Spark Configuration Tuning on EMR
You're running a Spark job on EMR with r5.2xlarge instances (8 vCPUs, 64 GB RAM). How do you calculate optimal executor configuration?
Spark Dynamic Allocation on EMR
Your EMR cluster runs multiple Spark jobs concurrently. Some jobs hog all resources while others wait. How do you enable fair resource sharing?
S3 Committers for Reliable Writes
Your Spark job on EMR occasionally produces missing or duplicate output files in S3. What's causing this and how do you fix it?
EMR Cluster Monitoring and Debugging
A Spark job on EMR failed after running for 3 hours. Walk me through how you would diagnose the failure.
EMR Security Configuration
Your compliance team requires encryption at rest and in transit, Kerberos authentication, and fine-grained access control on your EMR cluster. How do you configure this?
EMRFS vs HDFS: When to Use Each
Should you use HDFS or S3 (EMRFS) as the primary storage for Spark jobs on EMR? What are the trade-offs?
Troubleshooting Slow Spark Jobs on EMR
A Spark job that used to run in 30 minutes now takes 4 hours on the same EMR cluster. Data volume hasn't changed significantly. How do you diagnose and fix this?
EMR Cost Optimization Strategy
Your organization spends $50,000/month on EMR. Outline a comprehensive strategy to reduce costs by 40% without impacting SLAs.