Airflow
20 questionsDAG design, operators, XComs, scheduling, and troubleshooting. The Airflow questions you'll face in every DE interview.
Why Idempotent DAGs Matter
Why is idempotency critical in Airflow DAGs? Give an example of a non-idempotent task and how you would fix it.
TaskFlow API and XCom
Explain the TaskFlow API (`@task` decorator). How does it simplify XCom usage compared to the traditional approach?
BranchPythonOperator
How does `BranchPythonOperator` work? What happens to the tasks in branches that are not selected?
Execution Date vs Logical Date
What is the difference between `execution_date` and `logical_date` in Airflow? Why was the naming changed?
Schedule Interval vs Timetable
Compare `schedule_interval` with the timetable API. When would you use a custom timetable instead of a cron expression?
Dynamic DAGs
What are the common approaches for generating dynamic DAGs? What are the trade-offs of each?
XCom Push/Pull Under the Hood
How does XCom work under the hood? What are its limitations, and how do you handle large data transfers between tasks?
Variables vs Connections
What is the difference between Airflow Variables and Connections? When would you use each?
TriggerDagRunOperator and Cross-DAG Dependencies
How do you handle cross-DAG dependencies in Airflow? Compare `TriggerDagRunOperator`, `ExternalTaskSensor`, and dataset-driven scheduling.
Executor Types
Compare LocalExecutor, CeleryExecutor, and KubernetesExecutor. When would you choose each?
Backfill and Catchup
What is backfill in Airflow? How does the `catchup` parameter affect it?
KubernetesPodOperator
When would you use `KubernetesPodOperator` instead of `PythonOperator`? What are the key configuration options?
Pools for Resource Management
What are Airflow pools and when would you use them? How do they differ from concurrency settings?
Debugging Failed Tasks
A task in production has failed. Walk me through your debugging process in Airflow.
Custom Operators
When should you write a custom operator? Walk through the design of one.
SLA Monitoring
How do you implement SLA monitoring in Airflow? What are the limitations?
TaskGroup for DAG Organization
What is a TaskGroup and how does it differ from SubDAGs? Why were SubDAGs deprecated?
ExternalTaskSensor Pitfalls
What are common pitfalls when using `ExternalTaskSensor` and how do you mitigate them?
Airflow vs Prefect
Compare Airflow and Prefect. In what scenarios would you recommend one over the other?
Production Airflow Architecture
Design a production-grade Airflow deployment for a platform team serving 50+ data engineers writing 500+ DAGs. Cover architecture, scaling, CI/CD, and governance.