Python Shell vs Spark ETL Jobs
Your team needs to run a simple ETL job that reads a small CSV from S3, transforms a few columns, and writes to DynamoDB. A colleague suggests using a Glue Spark job. What would you recommend instead, and why?
Job Bookmarks for Incremental Processing
You have a Glue ETL job that processes new files landing in S3 daily. Currently it reprocesses everything on each run, taking 3 hours. How would you make it incremental?
DPU Sizing and Auto Scaling
Your Glue Spark job processes 500GB of Parquet data but keeps failing with OOM errors at 10 DPUs. How do you approach sizing and what Glue features help?
Crawlers vs Manual Catalog Tables
Your team debates whether to use Glue Crawlers or manually create Glue Data Catalog tables for a new data lake. What are the trade-offs?
Schema Evolution in Glue
Upstream producers start adding new columns to your JSON events without notice. Your Glue ETL pipeline breaks. How do you build a schema-evolution-resilient pipeline?
Pushdown Predicates vs Server-Side Filtering
Your Glue job reads from a partitioned S3 data lake but scans way more data than expected. Explain pushdown predicates and how they differ from regular Spark filters.
Handling Small Files Problem
Your Glue job writes output as thousands of small Parquet files (< 1MB each), causing downstream Athena queries to be slow and expensive. How do you fix this?
Glue Workflows and Triggers
You need to orchestrate 5 Glue jobs where Job B and C depend on Job A, and Job D depends on both B and C completing successfully. How do you set this up?
Glue Connection Pooling for JDBC
Your Glue job reads from an RDS PostgreSQL database but overwhelms it with connections, causing production issues. How do you control connection usage?
Glue DataBrew for Data Quality
A data analyst wants to profile and clean a messy CSV dataset but doesn't know Python or Spark. What Glue service would you recommend?
Glue vs EMR Decision
Your team is building a new data platform and debates between AWS Glue and EMR for ETL. What factors drive this decision, and when would you choose each?
Glue ETL Error Handling
Your Glue ETL job fails intermittently with various errors — bad records, S3 throttling, and occasional OOM. How do you build a resilient pipeline?
Glue Crawler Troubleshooting
A Glue crawler creates separate tables for what should be a single table, or merges tables that should be separate. How do you troubleshoot and fix this?
Glue Cost Optimization
Your organization's Glue bill has tripled in 3 months. You're asked to optimize costs without sacrificing pipeline reliability. Walk through your approach.
Glue Studio Visual ETL
A junior team member needs to create a Glue ETL job but isn't comfortable with PySpark code. What would you suggest?