Python
62 questionsPython data structures, concurrency (GIL, asyncio, multiprocessing), testing, file I/O, and real-world production patterns.
List vs Tuple vs Set vs Dict
Explain the differences between list, tuple, set, and dict in Python. When do you use each? What are the time complexities?
List Comprehension vs Generator
What's the difference between a list comprehension and a generator expression? When do you use each? Show memory implications.
Dictionaries Deep Dive
How does Python dict work internally? What makes a good dict key? What's the time complexity and when does it degrade?
collections Module
Show practical uses of Counter, defaultdict, deque, namedtuple, and OrderedDict. When would you reach for each?
Mutable Default Arguments
What's wrong with this code? Why does it produce unexpected results? ```python def add_item(item, items=[]): items.append(item) return items print(add_item('a')) # ['a'] print(add_item('b')) # ['a', 'b'] — why not ['b']? ```
Decorators
Write a decorator that: (1) logs function calls with args, (2) measures execution time, (3) retries on failure. Explain how decorators work internally.
GIL — Global Interpreter Lock
What is the GIL? Why does Python have it? How does it affect multithreading? How do you work around it for CPU-bound tasks?
asyncio Fundamentals
Explain asyncio in Python. When do you use it? Write an example that fetches 100 URLs concurrently with proper error handling.
Threading vs Multiprocessing vs asyncio
Your pipeline has 3 stages: (1) fetch data from 50 APIs, (2) CPU-heavy transformation, (3) write to 10 databases. Which concurrency model do you use for each stage? Why?
Race Conditions and Locks
This code has a race condition. Find it and fix it: ```python balance = 0 def deposit(amount): global balance current = balance time.sleep(0.01) # simulate processing balance = current + amount threads = [threading.Thread(target=deposit, args=(100,)) for _ in range(10)] for t in threads: t.start() for t in threads: t.join() print(balance) # Expected: 1000, Actual: 100-300 ```
Reading Large Files Efficiently
You need to process a 50GB CSV file that doesn't fit in memory. Show different approaches in pure Python and with pandas.
Context Managers
Write a custom context manager two ways: using a class and using @contextmanager. Show real-world examples beyond file handling.
Unit Testing with pytest
Write comprehensive tests for a function that processes orders: validates input, calculates tax, applies discounts, and returns total. Use pytest fixtures, parametrize, and mocking.
Python Memory Management
Your Python data pipeline is using 30GB of memory for what should be a 5GB dataset. Explain Python's memory model and how to diagnose/fix memory bloat.
Exception Handling Best Practices
What's wrong with this error handling code? Rewrite it properly. ```python try: data = fetch_data() processed = transform(data) save(processed) except: pass ```
Dataclasses vs NamedTuple vs Pydantic
When do you use dataclass, NamedTuple, and Pydantic model? Show code examples and explain the tradeoffs.
String Internals and Performance
Why is string concatenation in a loop slow? What's the efficient way to build strings? Explain string interning.
args and kwargs
Explain *args and **kwargs. When do you use each? What's the order of parameters in a function signature?
Python Packaging and Imports
Your data pipeline project has circular imports. Explain how Python imports work, what causes circular imports, and 3 ways to fix them.
Type Hints and mypy
Add comprehensive type hints to this code. How does typing help in data engineering? Show generics, Optional, Union, and callable types.
Closures and Scoping
What's a closure? What will this print and why? ```python funcs = [] for i in range(5): funcs.append(lambda: i) print([f() for f in funcs]) # What's the output? ```
Design Patterns in Python
Show Pythonic implementations of: Singleton, Factory, Observer, and Strategy patterns. How do Python features make them simpler than in Java?
Profiling and Optimization
Your Python ETL script takes 45 minutes. Before optimizing, how do you identify the bottleneck? Show profiling tools and optimization techniques.
itertools for Data Engineering
Show how itertools functions help in data processing: chain, groupby, islice, product, combinations, accumulate. Give real DE use cases.
Walrus Operator and Modern Python
Show 5 modern Python features (3.8+) that make code cleaner: walrus operator, match/case, f-string debugging, union types, and exception groups.
Working with APIs
Write production-quality code to fetch paginated data from a REST API with: rate limiting, retries with backoff, timeout handling, and connection pooling.
Python Logging Best Practices
Set up production logging for a data pipeline: structured JSON logs, log levels, rotation, and integration with monitoring systems.
Metaclasses and __init_subclass__
What are metaclasses? When would you actually use one in production? Show a practical example and explain simpler alternatives.
Environment and Dependency Management
How do you manage Python environments and dependencies in a production data pipeline? Compare venv, conda, poetry, and Docker.
Debugging Production Python
Your Python pipeline crashes in production at 3 AM with a cryptic error. You can't reproduce it locally. Walk through your debugging approach and tools.
Pandas DataFrame Basics
Create a DataFrame, select columns, filter rows, add columns, and sort. Show both loc/iloc and boolean indexing.
Merge, Join, Concat
Show all types of merges in pandas. How do you handle duplicate columns? What's the difference between merge, join, and concat?
GroupBy and Aggregation
Show advanced groupby patterns: multiple aggregations, custom agg functions, transform (broadcast back), filter groups, and named aggregation.
Handling Missing Data
Your dataset has NaN, None, empty strings, and 'N/A' values scattered across columns. Show a complete strategy for detecting, handling, and imputing missing data.
apply vs vectorize vs iterrows
Your colleague's pandas code uses iterrows() and takes 30 minutes. Explain the performance hierarchy and rewrite it using vectorized operations.
Pandas Memory Optimization
Your 10GB CSV loads into pandas as a 40GB DataFrame. Why? How do you reduce memory usage by 80% without losing data?
Time Series in Pandas
Show how to handle time series data: parse dates, set DatetimeIndex, resample, rolling windows, shift/lag, and time-based filtering.
Pivot, Melt, Stack
Transform data between wide and long formats. Show pivot, melt, stack/unstack, and cross-tabulation with real examples.
String Operations in Pandas
Clean messy string data: extract patterns with regex, split/combine columns, normalize text, and handle mixed types.
Window Functions in Pandas
Implement SQL-style window functions in pandas: rank within groups, running totals, lag/lead, first/last per group, and nth value.
Pandas vs Polars vs PySpark
When do you use pandas vs Polars vs PySpark? Show the same operation in all three. What are the performance differences?
Read/Write Performance
Show how to optimize pandas read/write for different formats. Compare CSV, Parquet, Feather, and JSON. When to use each?
MultiIndex and Advanced Indexing
Show how to create, query, and manipulate MultiIndex DataFrames. When are they useful? When should you avoid them?
NumPy Essentials for DE
Show NumPy operations that data engineers use daily: vectorized operations, boolean indexing, broadcasting, and where/select. Why is NumPy so much faster than Python lists?
Data Cleaning Pipeline
You receive a messy CSV from a vendor with: inconsistent column names, mixed date formats, duplicate rows, outliers, mixed types in numeric columns, and encoding issues. Build a complete cleaning pipeline.
merge_asof and Interval Joins
You have trades at exact timestamps and market quotes at different timestamps. Join each trade to the most recent quote BEFORE the trade time. Show merge_asof and explain when to use it.
Method Chaining and pipe()
Rewrite this step-by-step pandas code into clean method chains. When is method chaining good vs bad?
Regular Expressions for Data Cleaning
Write regex patterns to: extract emails, parse phone numbers, validate IPs, clean currency values, and extract structured data from unstructured text.
Date and Timezone Handling
Your data has timestamps from 3 timezones (UTC, IST, EST) stored as strings. Normalize everything to UTC, handle DST transitions, and calculate business-day differences.
SQLAlchemy for Data Pipelines
Show how to use SQLAlchemy in a data pipeline: connection pooling, bulk inserts, reading with pandas, and handling transactions. Compare Core vs ORM.
Garbage Collection and Reference Counting
Your long-running Python pipeline slowly leaks memory over 24 hours. Explain Python's garbage collection, how to find memory leaks, and common causes.
Pandas Cheat Sheet: SQL to Pandas
Translate these SQL queries to pandas: 1. SELECT with WHERE 2. GROUP BY with HAVING 3. JOIN with aggregation 4. Window function (ROW_NUMBER, LAG) 5. CASE WHEN 6. UNION ALL
Property-Based Testing with Hypothesis
What is property-based testing? When is it better than example-based testing? Write property tests for a data transformation function using Hypothesis.
Abstract Base Classes and Protocols
When do you use ABC vs Protocol in Python? Show how to design a pluggable data pipeline with interchangeable extractors, transformers, and loaders.
Pandas Performance Gotchas
List the top 10 pandas performance mistakes and their fixes. Show benchmarks for each.
Diamond Problem and MRO
Explain the diamond problem in Python multiple inheritance. What is MRO? What will this print? ```python class A: def greet(self): return 'A' class B(A): def greet(self): return 'B' class C(A): def greet(self): return 'C' class D(B, C): pass print(D().greet()) print(D.__mro__) ```
Dunder Methods (Magic Methods)
Implement a custom class that supports: len(), iteration, comparison, addition, string representation, context manager, and can be used as a dict key. Show all relevant dunder methods.
Composition vs Inheritance
Your team has a deep inheritance hierarchy (5 levels) for data pipeline classes. It's becoming unmaintainable. Refactor using composition. When is inheritance still the right choice?
Descriptors and Properties
What's a descriptor in Python? How does @property work under the hood? Write a descriptor that validates types and ranges on assignment.
Class Methods, Static Methods, Instance Methods
Explain the difference between instance methods, class methods, and static methods. When do you use each? Show the factory pattern with classmethod.
Garbage Collection Deep Dive
Explain Python's complete garbage collection system: reference counting, generational GC, and the gc module. When does GC fail? How to tune it for data pipelines?
Shallow Copy vs Deep Copy
What's the difference between assignment, shallow copy, and deep copy? Show how nested objects behave with each.