Kafka
30 questionsKafka architecture, exactly-once semantics, schema evolution, production troubleshooting, and event-driven design patterns.
Kafka Architecture Overview
Explain Kafka's core architecture: brokers, topics, partitions, producers, consumers, and consumer groups. Draw the relationship.
Partitions and Ordering
How does Kafka guarantee message ordering? If I produce messages A, B, C with the same key, will consumers always see them in that order?
Consumer Groups and Rebalancing
What happens during a consumer group rebalance? Why is it problematic? How do you minimize rebalance impact?
Offsets and Commit Strategies
Explain Kafka consumer offsets. What's the difference between auto-commit and manual commit? When does each cause problems?
Replication and ISR
Explain Kafka replication. What's the ISR (In-Sync Replicas)? What happens when a broker goes down? What's unclean leader election?
Producer Internals
Walk through what happens internally when a Kafka producer sends a message. Include batching, compression, partitioning, and acknowledgment.
Log Compaction
What is log compaction in Kafka? How is it different from time-based retention? When would you use it?
Kafka vs Other Message Systems
When would you choose Kafka vs RabbitMQ vs SQS vs Kinesis? What makes Kafka different?
Choosing Partition Count
Your team is creating a new Kafka topic for order events. How do you choose the number of partitions? The system handles 50K orders/second at peak.
Consumer Lag Monitoring
What is consumer lag? How do you monitor it? Your consumer lag is growing — walk through your debugging process.
KRaft vs ZooKeeper
What is KRaft mode? Why is Kafka removing ZooKeeper? What changes for operators?
Exactly-Once Semantics
Explain Kafka's exactly-once semantics end-to-end. What are the three levels of guarantee? How do transactions work?
Idempotent Producer Deep Dive
How does the idempotent producer work internally? What does it prevent? What doesn't it prevent?
Schema Registry and Avro
What is Confluent Schema Registry? Why use Avro/Protobuf over JSON? How does schema evolution work?
Handling Schema Changes in Production
Your producer team needs to rename a field and change a field type (int → long). Both are breaking changes. How do you handle this without downtime?
Kafka Connect
What is Kafka Connect? When would you use it instead of writing a custom producer/consumer? Show a real-world example.
Event-Driven Architecture with Kafka
Design an event-driven order processing system using Kafka. An order goes through: placed → payment → inventory → shipping → delivered. How do you handle failures at each stage?
Kafka Streams vs Flink vs Spark Streaming
When would you choose Kafka Streams vs Flink vs Spark Structured Streaming? What are the key architectural differences?
Outbox Pattern
Your service needs to update a database AND publish a Kafka event atomically. A crash between the two operations causes inconsistency. How do you solve this?
Data Loss Scenarios
List all the ways data can be lost in Kafka. For each, explain the cause and the configuration to prevent it.
Message Too Large / Poison Pill
Your consumer keeps crashing on a specific message. It restarts, re-reads the same message, crashes again — infinite loop. How do you handle this 'poison pill' scenario?
Kafka Cluster Sizing
You're designing a Kafka cluster for 500K messages/sec, 1KB average message, 7-day retention, replication factor 3. How many brokers, what disk, what memory?
Slow Consumer Troubleshooting
Your consumer group processes messages too slowly. Lag is growing by 10K messages/minute. Walk through your investigation and fixes.
Cross-DC Replication
Your company needs Kafka data replicated across two data centers for disaster recovery. What are the options? What are the trade-offs?
Kafka Security
How do you secure a Kafka cluster? Cover authentication, authorization, encryption, and audit.
Message Ordering Across Services
You have 3 microservices that produce events to the same Kafka topic: order-created, payment-confirmed, order-shipped. A consumer sometimes receives payment-confirmed before order-created. Why? How do you fix it?
Kafka Connect Failures
Your Debezium CDC source connector stops capturing changes from PostgreSQL. The connector shows as RUNNING but no new messages appear. What happened?
Backpressure Handling
Your Kafka producer is getting BufferExhaustedException. What's happening and how do you handle backpressure?
Kafka for Event Sourcing
Can Kafka be used as an event store for event sourcing? What are the benefits and limitations? When should you use a dedicated event store instead?
Migrate from Single Topic to Multi-Topic
Your team made all events go to one Kafka topic (events). It now has 500 partitions but consumers are struggling because payment events must be processed separately from analytics events. How do you split it?