Skip to main content
Mapagam

56 SECTIONS · 1174 TOPICS

Kafka Roadmap

  1. Understanding Kafka Architecture

    1. 1. Understanding Broker Role
    2. Example: Connecting to brokers
    3. 2. Understanding Topics and Partitions
    4. Example: Creating a topic with partitions
    5. 3. Understanding Producers and Consumers
    6. Example: Minimal produce and consume
    7. 4. Understanding Consumer Groups
    8. Example: Assigning a consumer group
    9. 5. Understanding Replication
    10. Example: Inspecting replication
    11. 6. Understanding Log Segments
    12. Example: On-disk segment files
    13. 7. Understanding Offsets
    14. Example: Reading current offset
    15. 8. Understanding Message Flow
    16. Example: End-to-end flow with ack
    17. 9. Understanding ZooKeeper Role
    18. Example: Legacy ZooKeeper connection
    19. 10. Understanding KRaft Architecture
    20. Example: KRaft controller config

  2. Installing and Configuring Kafka

    1. 1. Downloading Kafka
    2. Example: Download and extract
    3. 2. Setting Up Java Requirements
    4. Example: Verify Java
    5. 3. Configuring Broker Properties
    6. Example: Core server.properties
    7. 4. Setting Broker ID
    8. Example: Set node id
    9. 5. Configuring Log Directories
    10. Example: Multiple log dirs
    11. 6. Configuring Listeners
    12. Example: Internal and external listeners
    13. 7. Setting Up Cluster ID
    14. Example: Generate and format storage
    15. 8. Starting ZooKeeper Server
    16. Example: Start ZooKeeper (legacy)
    17. 9. Starting Kafka Broker
    18. Example: Start broker
    19. 10. Using KRaft Mode
    20. Example: Single-node KRaft startup
    21. 11. Verifying Installation
    22. Example: Smoke test

  3. Managing Topics

    1. 1. Creating Topics
    2. Example: Create with configs
    3. 2. Listing Topics
    4. Example: List topics
    5. 3. Describing Topics
    6. Example: Describe a topic
    7. 4. Setting Partition Count
    8. Example: Add partitions
    9. 5. Setting Replication Factor
    10. Example: Increase RF via reassignment
    11. 6. Altering Topic Configuration
    12. Example: Alter retention
    13. 7. Deleting Topics
    14. Example: Delete a topic
    15. 8. Configuring Topic Retention
    16. Example: Time + size retention
    17. 9. Setting Cleanup Policy
    18. Example: Enable compaction
    19. 10. Configuring Segment Settings
    20. Example: Smaller segments for faster compaction

  4. Working with Partitions

    1. 1. Understanding Partition Assignment
    2. Example: Keyed assignment
    3. 2. Specifying Partition Key
    4. Example: Record with key
    5. 3. Viewing Partition Distribution
    6. Example: Find under-replicated partitions
    7. 4. Adding Partitions
    8. Example: Grow partitions
    9. 5. Understanding Leader Election
    10. Example: Trigger preferred election
    11. 6. Understanding Log Segments
    12. Example: Dump a segment
    13. 7. Configuring Preferred Leader Election
    14. Example: Enable auto rebalance
    15. 8. Configuring Min In-Sync Replicas
    16. Example: Set minimum ISR
    17. 9. Setting Partition Assignment Strategy
    18. Example: Cooperative sticky assignor
    19. 10. Manual Partition Reassignment
    20. Example: Execute reassignment

  5. Producing Messages

    1. 1. Creating Producer Instance
    2. Example: Construct a producer
    3. 2. Setting Producer Properties
    4. Example: Durable producer config
    5. 3. Configuring Bootstrap Servers
    6. Example: Multiple bootstrap brokers
    7. 4. Configuring Serializers
    8. Example: Avro value serializer
    9. 5. Sending Messages
    10. Example: Send a record
    11. 6. Using Synchronous Send
    12. Example: Synchronous send
    13. 7. Using Asynchronous Send
    14. Example: Async with callback
    15. 8. Setting Message Key
    16. Example: Keyed record
    17. 9. Specifying Target Partition
    18. Example: Send to specific partition
    19. 10. Adding Message Headers
    20. Example: Add headers
    21. 11. Configuring Acknowledgments
    22. Example: Strongest durability
    23. 12. Flushing Producer Buffer
    24. Example: Flush before exit

  6. Configuring Producer Settings

    1. 1. Setting Batch Size
    2. Example: Larger batches
    3. 2. Configuring Linger Time
    4. Example: Batch for throughput
    5. 3. Setting Buffer Memory
    6. Example: Increase buffer
    7. 4. Configuring Compression
    8. Example: Enable zstd
    9. 5. Setting Max Request Size
    10. Example: Allow larger messages
    11. 6. Setting Request Timeout
    12. Example: Set request timeout
    13. 7. Configuring Delivery Timeout
    14. Example: Cap total delivery time
    15. 8. Configuring Retries
    16. Example: Retry config
    17. 9. Setting Idempotence
    18. Example: Idempotent producer
    19. 10. Configuring Max In-Flight Requests
    20. Example: Set in-flight limit

  7. Consuming Messages

    1. 1. Creating Consumer Instance
    2. Example: Construct a consumer
    3. 2. Setting Consumer Properties
    4. Example: Manual-commit config
    5. 3. Configuring Bootstrap Servers
    6. Example: Bootstrap servers
    7. 4. Configuring Deserializers
    8. Example: Avro deserializer
    9. 5. Setting Consumer Group
    10. Example: Static membership
    11. 6. Subscribing to Topics
    12. Example: Subscribe to topics
    13. 7. Polling Messages
    14. Example: Poll loop
    15. 8. Processing Records
    16. Example: Access record fields
    17. 9. Using Manual Assignment
    18. Example: Manual assign
    19. 10. Seeking to Position
    20. Example: Seek to offset
    21. 11. Closing Consumer
    22. Example: Graceful shutdown

  8. Configuring Consumer Settings

    1. 1. Setting Fetch Size
    2. Example: Batch-oriented fetch
    3. 2. Configuring Fetch Wait Time
    4. Example: Reduce fetch latency
    5. 3. Setting Max Poll Records
    6. Example: Limit poll batch
    7. 4. Setting Max Partition Fetch Bytes
    8. Example: Increase per-partition fetch
    9. 5. Configuring Partition Assignment Strategy
    10. Example: Set strategy
    11. 6. Setting Auto Offset Reset
    12. Example: Process from beginning
    13. 7. Configuring Isolation Level
    14. Example: Read committed only
    15. 8. Setting Session Timeout
    16. Example: Tune session timeout
    17. 9. Setting Heartbeat Interval
    18. Example: Heartbeat config
    19. 10. Configuring Max Poll Interval
    20. Example: Allow longer processing
    21. 11. Configuring Check CRCs
    22. Example: CRC check setting
    23. 12. Setting Client ID
    24. Example: Set client id

  9. Managing Consumer Groups

    1. 1. Understanding Group Coordinator
    2. Example: Find coordinator
    3. 2. Understanding Rebalancing
    4. Example: Rebalance listener
    5. 3. Listing Consumer Groups
    6. Example: List groups
    7. 4. Describing Consumer Groups
    8. Example: Describe a group
    9. 5. Viewing Consumer Lag
    10. Example: Check lag
    11. 6. Resetting Offsets
    12. Example: Reset to earliest
    13. 7. Deleting Consumer Groups
    14. Example: Delete a group
    15. 8. Configuring Session Timeout
    16. Example: Broker bounds
    17. 9. Setting Heartbeat Interval
    18. Example: Align heartbeat
    19. 10. Configuring Max Poll Interval
    20. Example: Prevent eviction during heavy work

  10. Managing Offsets

    1. 1. Understanding Offset Commits
    2. Example: Commit after processing
    3. 2. Enabling Auto Commit
    4. Example: Enable auto commit
    5. 3. Setting Auto Commit Interval
    6. Example: Frequent auto commit
    7. 4. Committing Synchronously
    8. Example: Synchronous commit
    9. 5. Committing Asynchronously
    10. Example: Async with final sync
    11. 6. Committing Specific Offsets
    12. Example: Commit explicit offsets
    13. 7. Storing Offsets Externally
    14. Example: Restore offset from DB
    15. 8. Resetting to Earliest
    16. Example: Seek to beginning
    17. 9. Resetting to Latest
    18. Example: Skip to latest
    19. 10. Resetting to Specific Offset
    20. Example: Reset to offset 1000
    21. 11. Resetting by Timestamp
    22. Example: Reset by datetime

  11. Working with Message Serialization

    1. 1. Understanding Serialization Concept
    2. Example: Serializer interface
    3. 2. Using String Serializer
    4. Example: String serializer
    5. 3. Using Byte Array Serializer
    6. Example: Byte array producer
    7. 4. Using Integer Serializer
    8. Example: Integer key
    9. 5. Using Long Serializer
    10. Example: Long timestamp value
    11. 6. Using JSON Serialization
    12. Example: Custom JSON serializer
    13. 7. Using Avro Serialization
    14. Example: Avro producer
    15. 8. Using Protobuf Serialization
    16. Example: Protobuf serializer
    17. 9. Creating Custom Serializers
    18. Example: Register custom serializer
    19. 10. Handling Serialization Errors
    20. Example: Safe deserialization wrapper
    21. 11. Configuring Schema Registry
    22. Example: Registry client config

  12. Understanding Message Compression

    1. 1. Understanding Compression Benefits
    2. Example: Enable producer compression
    3. 2. Using GZIP Compression
    4. Example: GZIP
    5. 3. Using Snappy Compression
    6. Example: Snappy
    7. 4. Using LZ4 Compression
    8. Example: LZ4
    9. 5. Using ZSTD Compression
    10. Example: ZSTD
    11. 6. Configuring Compression Level
    12. Example: Tune zstd level
    13. 7. Understanding Compression Ratio
    14. Example: Monitor compression ratio
    15. 8. Comparing Compression Performance
    16. LZ4 / Snappy
    17. GZIP / ZSTD
    18. Example: Benchmark settings
    19. 9. Setting Producer Compression
    20. Example: Producer compression
    21. 10. Broker Compression Handling
    22. Example: Broker compression config
    23. 11. Monitoring Compression Metrics
    24. Example: Read compression metric

  13. Working with Message Headers

    1. 1. Understanding Headers Concept
    2. Example: Headers interface
    3. 2. Adding Headers to Producer
    4. Example: Add producer headers
    5. 3. Reading Headers in Consumer
    6. Example: Read a header
    7. 4. Setting Custom Headers
    8. Example: Multiple custom headers
    9. 5. Using Headers for Routing
    10. Example: Route by header
    11. 6. Storing Metadata in Headers
    12. Example: Retry count header
    13. 7. Filtering by Headers
    14. Example: Filter by header
    15. 8. Understanding Header Serialization
    16. Example: Encode/decode header value
    17. 9. Propagating Headers in Streams
    18. Example: Headers in processor
    19. 10. Adding Trace IDs
    20. Example: Inject trace id

  14. Understanding Message Timestamps

    1. 1. Understanding Timestamp Types
    2. Example: Read record timestamp
    3. 2. Using Create Time
    4. Example: Explicit create time
    5. 3. Using Log Append Time
    6. Example: Set log append time
    7. 4. Configuring Timestamp Type
    8. Example: Broker default
    9. 5. Setting Producer Timestamp
    10. Example: Backdated event
    11. 6. Understanding Timestamp Ordering
    12. Example: Detect out-of-order
    13. 7. Using Timestamps for Retention
    14. Example: Retention with append time
    15. 8. Filtering by Timestamp
    16. Example: Seek by timestamp
    17. 9. Configuring Timestamp Difference
    18. Example: Limit timestamp skew
    19. 10. Resetting Offsets by Timestamp
    20. Example: Reset group to datetime

  15. Understanding Message Ordering

    1. 1. Understanding Ordering Guarantees
    2. Example: Per-key ordering
    3. 2. Preserving Order with Single Partition
    4. Example: Single-partition topic
    5. 3. Using Message Keys for Ordering
    6. Example: Key-based ordering
    7. 4. Configuring Max In-Flight Requests
    8. Example: Safe in-flight + idempotence
    9. 5. Enabling Idempotent Producer
    10. Example: Idempotent producer
    11. 6. Understanding Ordering Across Partitions
    12. Example: Reorder by timestamp buffer
    13. 7. Implementing Total Ordering
    14. Example: Embed sequence number
    15. 8. Handling Out-of-Order Messages
    16. Example: Streams grace period
    17. 9. Using Timestamps for Ordering
    18. Example: Custom timestamp extractor
    19. 10. Configuring Retries for Ordering
    20. Example: Order-preserving retries

  16. Understanding Replication

    1. 1. Understanding Replication Concept
    2. Example: Topic with RF=3
    3. 2. Understanding Leader and Followers
    4. Example: Identify leader
    5. 3. Configuring Replication Factor
    6. Example: Broker default RF
    7. 4. Understanding In-Sync Replicas
    8. Example: Check ISR
    9. 5. Configuring Replica Lag
    10. Example: Tune replica lag window
    11. 6. Understanding Leader Election
    12. Example: Disable unclean election
    13. 7. Understanding Replica High Watermark
    14. Example: HW limits reads
    15. 8. Understanding Log End Offset
    16. Example: Inspect LEO via JMX
    17. 9. Configuring Preferred Leader
    18. Example: Run preferred election
    19. 10. Understanding Replica Fetcher
    20. Example: More fetcher threads
    21. 11. Manual Replica Reassignment
    22. Example: Reassign replicas

  17. Implementing Log Compaction

    1. 1. Understanding Compaction Concept
    2. Example: Compacted topic concept
    3. 2. Enabling Log Compaction
    4. Example: Enable compaction
    5. 3. Understanding Compaction Process
    6. Example: Cleaner status in logs
    7. 4. Understanding Tombstone Messages
    8. Example: Send a tombstone
    9. 5. Configuring Compaction Lag
    10. Example: Set compaction lag
    11. 6. Setting Delete Retention
    12. Example: Extend delete retention
    13. 7. Configuring Segment Size
    14. Example: Smaller segments for compaction
    15. 8. Setting Min Cleanable Ratio
    16. Example: Aggressive cleaning
    17. 9. Configuring Compaction Thread
    18. Example: More cleaner threads
    19. 10. Setting Compaction Buffer
    20. Example: Larger dedupe buffer
    21. 11. Monitoring Compaction Progress
    22. Example: Cleaner JMX bean

  18. Understanding Message Delivery Semantics

    1. 1. Understanding Delivery Guarantees
    2. Example: At-least-once
    3. 2. Understanding At-Most-Once Delivery
    4. Example: At-most-once
    5. 3. Understanding At-Least-Once Delivery
    6. Example: At-least-once with idempotent sink
    7. 4. Implementing Exactly-Once Semantics
    8. Example: Streams exactly-once
    9. 5. Configuring Producer Idempotence
    10. Example: Idempotent config
    11. 6. Using Transactional Producer
    12. Example: Transactional send
    13. 7. Setting Consumer Isolation Level
    14. Example: Read committed
    15. 8. Understanding Duplicate Messages
    16. Example: Dedup by key+offset
    17. 9. Handling Message Ordering
    18. Example: Ordered EOS pipeline
    19. 10. Implementing Idempotent Consumer
    20. Example: Atomic DB upsert + offset
    21. 11. Understanding Message Loss Scenarios
    22. Example: No-loss producer/topic

  19. Implementing Transactions

    1. 1. Understanding Transactional Concept
    2. Example: Transactional pipeline shape
    3. 2. Enabling Transactional Producer
    4. Example: Configure transactional id
    5. 3. Initializing Transactions
    6. Example: Init transactions
    7. 4. Beginning Transaction
    8. Example: Begin a transaction
    9. 5. Sending Transactional Messages
    10. Example: Multi-topic atomic write
    11. 6. Sending Offsets in Transaction
    12. Example: Offsets within transaction
    13. 7. Committing Transaction
    14. Example: Commit
    15. 8. Aborting Transaction
    16. Example: Abort on failure
    17. 9. Configuring Transaction Timeout
    18. Example: Set transaction timeout
    19. 10. Setting Read Committed
    20. Example: Consumer read committed
    21. 11. Understanding Exactly-Once Semantics
    22. Example: Streams EOS

  20. Implementing Retry Mechanisms

    1. 1. Understanding Retry Behavior
    2. Example: Retry config
    3. 2. Configuring Retry Count
    4. Example: Limit retries
    5. 3. Setting Retry Backoff
    6. Example: Backoff config
    7. 4. Configuring Delivery Timeout
    8. Example: Set delivery deadline
    9. 5. Understanding Retriable Exceptions
    10. Example: Check retriable
    11. 6. Handling Non-Retriable Errors
    12. Example: Route non-retriable to DLQ
    13. 7. Implementing Exponential Backoff
    14. Example: Consumer-side backoff
    15. 8. Using Circuit Breaker Pattern
    16. Example: Circuit breaker around sink
    17. 9. Configuring Request Timeout
    18. Example: Request timeout
    19. 10. Implementing Consumer Retry Logic
    20. Example: Retry-topic pattern

  21. Configuring Broker Settings

    1. 1. Setting Broker ID
    2. Example: Broker id
    3. 2. Configuring Listeners
    4. Example: Listener config
    5. 3. Setting Log Directories
    6. Example: Multiple disks
    7. 4. Configuring Number of Threads
    8. Example: Thread tuning
    9. 5. Setting Socket Buffer Sizes
    10. Example: Larger socket buffers
    11. 6. Configuring Max Message Size
    12. Example: Allow large messages
    13. 7. Setting Default Replication Factor
    14. Example: Replication defaults
    15. 8. Configuring Min ISR
    16. Example: Default min ISR
    17. 9. Setting Unclean Leader Election
    18. Example: Keep unclean disabled
    19. 10. Configuring Log Retention
    20. Example: Retention defaults
    21. 11. Setting Log Segment Size
    22. Example: Segment size
    23. 12. Configuring Log Cleanup
    24. Example: Cleanup policy

  22. Using KRaft Consensus Protocol

    1. 1. Understanding KRaft Architecture
    2. Example: KRaft roles
    3. 2. Understanding Metadata Replication
    4. Example: Inspect metadata log
    5. 3. Configuring Metadata Log
    6. Example: Dedicated metadata dir
    7. 4. Setting Controller Quorum Voters
    8. Example: Three voters
    9. 5. Formatting Storage
    10. Example: Format storage
    11. 6. Starting KRaft Controller
    12. Example: Start controller
    13. 7. Starting KRaft Broker
    14. Example: Start broker
    15. 8. Using Combined Mode
    16. Example: Combined roles
    17. 9. Configuring Controller Listener
    18. Example: Controller listener
    19. 10. Migrating from ZooKeeper
    20. Example: Enable migration

  23. Managing Cluster Metadata

    1. 1. Understanding Metadata Topics
    2. Example: List internal topics
    3. 2. Viewing Cluster Metadata
    4. Example: Metadata shell
    5. 3. Describing Cluster Configuration
    6. Example: Describe cluster config
    7. 4. Managing Broker Configuration
    8. Example: Dynamic broker change
    9. 5. Understanding Controller Role
    10. Example: Find active controller
    11. 6. Monitoring Controller Metrics
    12. Example: JMX object name
    13. 7. Handling Metadata Updates
    14. Example: Replication lag
    15. 8. Configuring Metadata Max Age
    16. Example: Client metadata age
    17. 9. Understanding Partition Metadata
    18. Example: Describe partition metadata
    19. 10. Debugging Metadata Issues
    20. Example: Dump metadata records

  24. Using Kafka Streams

    1. 1. Understanding Streams Concept
    2. Example: Streams dependency
    3. 2. Creating Streams Application
    4. Example: App skeleton
    5. 3. Building Stream Topology
    6. Example: Describe topology
    7. 4. Transforming Streams
    8. Example: Map and filter
    9. 5. Aggregating Streams
    10. Example: Count per key
    11. 6. Joining Streams
    12. Example: Stream-table join
    13. 7. Windowing Operations
    14. Example: Windowed count
    15. 8. Using State Stores
    16. Example: Query a store
    17. 9. Processing Records
    18. Example: Peek for logging
    19. 10. Using Processors API
    20. Example: Processor API
    21. 11. Starting Streams Application
    22. Example: Start with hook
    23. 12. Handling Streams Exceptions
    24. Example: Replace failed thread

  25. Configuring Kafka Streams

    1. 1. Setting Application ID
    2. Example: Application id
    3. 2. Configuring Bootstrap Servers
    4. Example: Bootstrap servers
    5. 3. Setting Default Serdes
    6. Example: Default serdes
    7. 4. Configuring Commit Interval
    8. Example: Commit interval
    9. 5. Setting Cache Size
    10. Example: Cache size
    11. 6. Configuring State Directory
    12. Example: State dir
    13. 7. Setting Processing Guarantee
    14. Example: Exactly-once
    15. 8. Configuring Replication Factor
    16. Example: Internal topic RF
    17. 9. Setting Number of Threads
    18. Example: Stream threads
    19. 10. Configuring Window Store Retention
    20. Example: Window retention

  26. Working with Time-Based Processing

    1. 1. Understanding Time Semantics
    2. Example: Time concept
    3. 2. Understanding Event Time
    4. Example: Default extractor
    5. 3. Understanding Processing Time
    6. Example: Wall-clock extractor
    7. 4. Understanding Ingestion Time
    8. Example: Set ingestion time
    9. 5. Configuring Timestamp Extractors
    10. Example: Per-stream extractor
    11. 6. Implementing Custom Timestampers
    12. Example: Custom extractor
    13. 7. Using Tumbling Windows
    14. Example: Tumbling window
    15. 8. Using Hopping Windows
    16. Example: Hopping window
    17. 9. Using Sliding Windows
    18. Example: Sliding window
    19. 10. Using Session Windows
    20. Example: Session window
    21. 11. Configuring Grace Period
    22. Example: Window with grace
    23. 12. Handling Out-of-Order Events
    24. Example: Suppress until close

  27. Implementing Stream Joins

    1. 1. Understanding Join Types
    2. Example: Join overview
    3. 2. Using Stream-Stream Join
    4. Example: Stream-stream join
    5. 3. Using Stream-Table Join
    6. Example: Enrich orders
    7. 4. Using Table-Table Join
    8. Example: Table-table join
    9. 5. Implementing Inner Join
    10. Example: Inner join
    11. 6. Implementing Left Join
    12. Example: Left join with default
    13. 7. Implementing Outer Join
    14. Example: Outer join
    15. 8. Configuring Join Windows
    16. Example: Asymmetric window
    17. 9. Setting Join Retention
    18. Example: Join store config
    19. 10. Understanding Join Semantics
    20. Example: GlobalKTable join
    21. 11. Optimizing Join Performance
    22. Example: Avoid repartition

  28. Using Kafka Connect

    1. 1. Understanding Connect Framework
    2. Example: Connect roles
    3. 2. Running Standalone Mode
    4. Example: Start standalone
    5. 3. Running Distributed Mode
    6. Example: Start distributed
    7. 4. Creating Source Connectors
    8. Example: JDBC source
    9. 5. Creating Sink Connectors
    10. Example: S3 sink
    11. 6. Configuring Connector Properties
    12. Example: SMT transform
    13. 7. Managing Connectors via REST API
    14. Example: Create via REST
    15. 8. Listing Connectors
    16. Example: List connectors
    17. 9. Checking Connector Status
    18. Example: Status check
    19. 10. Pausing and Resuming Connectors
    20. Example: Pause connector
    21. 11. Deleting Connectors
    22. Example: Delete connector

  29. Configuring Kafka Connect

    1. 1. Setting Bootstrap Servers
    2. Example: Bootstrap
    3. 2. Configuring Key and Value Converters
    4. Example: Converters
    5. 3. Setting Offset Storage
    6. Example: Offset topic
    7. 4. Configuring Config Storage
    8. Example: Config topic
    9. 5. Setting Status Storage
    10. Example: Status topic
    11. 6. Configuring Replication Factor
    12. Example: RF for all internal topics
    13. 7. Setting Group ID
    14. Example: Group id
    15. 8. Configuring Plugin Path
    16. Example: Plugin path
    17. 9. Setting REST Port
    18. Example: REST listener
    19. 10. Configuring Internal Topics
    20. Example: Pre-create config topic

  30. Using Schema Registry

    1. 1. Understanding Schema Registry Role
    2. Example: Wire format
    3. 2. Installing Schema Registry
    4. Example: Start registry
    5. 3. Configuring Schema Registry URL
    6. Example: Client config
    7. 4. Registering Schemas
    8. Example: Register schema
    9. 5. Retrieving Schemas
    10. Example: Get latest
    11. 6. Listing Subjects
    12. Example: List subjects
    13. 7. Deleting Schema Versions
    14. Example: Delete a version
    15. 8. Setting Compatibility Mode
    16. Example: Set subject compatibility
    17. 9. Checking Compatibility
    18. Example: Check compatibility
    19. 10. Exporting Schemas
    20. Example: Export all subjects

  31. Managing Schema Evolution

    1. 1. Understanding Schema Compatibility
    2. Example: Field with default
    3. 2. Implementing Backward Compatibility
    4. Example: Backward-safe add
    5. 3. Implementing Forward Compatibility
    6. Example: Forward-safe add
    7. 4. Using Full Compatibility
    8. Example: Full mode
    9. 5. Using Transitive Compatibility
    10. Example: Transitive mode
    11. 6. Configuring Compatibility Level
    12. Example: Global default
    13. 7. Handling Schema Evolution Errors
    14. Example: Pre-deploy check
    15. 8. Versioning Schemas
    16. Example: List versions
    17. 9. Using Schema References
    18. Example: Schema with reference
    19. 10. Migrating Schema Formats
    20. Example: Set schema type

  32. Implementing Custom Partitioners

    1. 1. Understanding Partitioner Concept
    2. Example: Interface
    3. 2. Creating Custom Partitioner
    4. Example: Class skeleton
    5. 3. Implementing partition() Method
    6. Example: partition()
    7. 4. Configuring Partitioner Class
    8. Example: Register partitioner
    9. 5. Using Hash-Based Partitioning
    10. Example: Murmur2 hash
    11. 6. Implementing Round-Robin Partitioning
    12. Example: Round-robin
    13. 7. Creating Sticky Partitioner
    14. Example: Built-in sticky behavior
    15. 8. Implementing Range Partitioner
    16. Example: Range mapping
    17. 9. Using Composite Key Partitioning
    18. Example: Tenant prefix routing
    19. 10. Handling Null Keys
    20. Example: Null-key fallback

  33. Implementing Custom Interceptors

    1. 1. Understanding Interceptor Concept
    2. Example: Interfaces
    3. 2. Creating Producer Interceptor
    4. Example: Producer interceptor
    5. 3. Creating Consumer Interceptor
    6. Example: Consumer interceptor
    7. 4. Implementing onSend() Method
    8. Example: Add header in onSend
    9. 5. Implementing onConsume() Method
    10. Example: Count in onConsume
    11. 6. Configuring Interceptor Chain
    12. Example: Register chain
    13. 7. Adding Monitoring Metrics
    14. Example: Track acks
    15. 8. Implementing Tracing
    16. Example: Inject traceparent
    17. 9. Adding Security Checks
    18. Example: Redact in onSend
    19. 10. Transforming Messages
    20. Example: Enrich value

  34. Using Admin API

    1. 1. Creating Admin Client
    2. Example: Create admin client
    3. 2. Creating Topics
    4. Example: Create topic
    5. 3. Listing Topics
    6. Example: List topics
    7. 4. Describing Topics
    8. Example: Describe topic
    9. 5. Deleting Topics
    10. Example: Delete topic
    11. 6. Creating Partitions
    12. Example: Add partitions
    13. 7. Altering Configs
    14. Example: Alter topic config
    15. 8. Describing Configs
    16. Example: Describe config
    17. 9. Describing Consumer Groups
    18. Example: Describe group
    19. 10. Listing Consumer Groups
    20. Example: List groups
    21. 11. Deleting Consumer Groups
    22. Example: Delete group
    23. 12. Deleting Records
    24. Example: Delete records

  35. Using Console Tools

    1. 1. Producing Messages
    2. Example: Console produce
    3. 2. Consuming Messages
    4. Example: Console consume
    5. 3. Reading from Beginning
    6. Example: From beginning
    7. 4. Specifying Consumer Group
    8. Example: With group
    9. 5. Setting Message Format
    10. Example: Print key and value
    11. 6. Using Key Deserializer
    12. Example: Long key deserializer
    13. 7. Setting Partition
    14. Example: Specific partition
    15. 8. Reading Specific Offset
    16. Example: Read from offset 500
    17. 9. Displaying Message Timestamp
    18. Example: Show metadata
    19. 10. Using Timeout
    20. Example: Timeout

  36. Using Kafka CLI Tools

    1. 1. Managing Topics
    2. Example: kafka-topics
    3. 2. Working with Consumer Groups
    4. Example: kafka-consumer-groups
    5. 3. Managing ACLs
    6. Example: kafka-acls
    7. 4. Managing Configs
    8. Example: kafka-configs
    9. 5. Reassigning Partitions
    10. Example: Reassign partitions
    11. 6. Preferred Leader Election
    12. Example: Trigger election
    13. 7. Verifying Consumer Group
    14. Example: Check lag
    15. 8. Verifying Producer
    16. Example: Producer perf test
    17. 9. Dumping Log Segments
    18. Example: Dump log
    19. 10. Mirroring Data
    20. Example: Run MirrorMaker 2
    21. 11. Running Performance Tests
    22. Example: Consumer perf test

  37. Implementing Security

    1. 1. Understanding Kafka Security Model
    2. Example: Security protocol
    3. 2. Enabling SSL Encryption
    4. Example: Enable SSL
    5. 3. Configuring SSL Certificates
    6. Example: Certificate config
    7. 4. Setting Keystore Password
    8. Example: Externalized password
    9. 5. Configuring SSL Endpoint Identification
    10. Example: Hostname verification
    11. 6. Enabling SASL Authentication
    12. Example: Enable SASL_SSL
    13. 7. Configuring SASL/PLAIN
    14. Example: SASL/PLAIN JAAS
    15. 8. Using SASL/SCRAM
    16. Example: Create SCRAM user
    17. 9. Configuring SASL/GSSAPI
    18. Example: GSSAPI JAAS
    19. 10. Setting Up SASL/OAUTHBEARER
    20. Example: OAuth config
    21. 11. Enabling Inter-Broker Security
    22. Example: Inter-broker SSL

  38. Managing Access Control

    1. 1. Understanding ACL Model
    2. Example: ACL anatomy
    3. 2. Enabling Authorization
    4. Example: Enable authorizer
    5. 3. Creating ACLs
    6. Example: Grant produce
    7. 4. Listing ACLs
    8. Example: List ACLs for topic
    9. 5. Removing ACLs
    10. Example: Remove ACL
    11. 6. Setting Principal Format
    12. Example: Principal mapping rule
    13. 7. Configuring Super Users
    14. Example: Super users
    15. 8. Understanding ACL Operations
    16. Example: Consumer needs Read on group
    17. 9. Setting Resource Patterns
    18. Example: Prefixed ACL
    19. 10. Using Wildcard ACLs
    20. Example: All-topics read
    21. 11. Configuring ACL Binding
    22. Example: ACL via Admin API

  39. Configuring Quotas

    1. 1. Understanding Quotas Concept
    2. Example: Quota throttling
    3. 2. Setting Producer Quotas
    4. Example: Producer quota
    5. 3. Configuring Consumer Quotas
    6. Example: Consumer quota
    7. 4. Setting Request Quotas
    8. Example: Request quota
    9. 5. Configuring Client-Level Quotas
    10. Example: Client.id quota
    11. 6. Setting User-Level Quotas
    12. Example: User quota
    13. 7. Using Default Quotas
    14. Example: Default user quota
    15. 8. Configuring Quota Window
    16. Example: Quota window
    17. 9. Monitoring Quota Violations
    18. Example: Throttle metric
    19. 10. Dynamic Quota Configuration
    20. Example: Remove a quota

  40. Implementing Dead Letter Queues

    1. 1. Understanding DLQ Pattern
    2. Example: DLQ flow
    3. 2. Creating DLQ Topic
    4. Example: Create DLQ
    5. 3. Routing Failed Messages
    6. Example: Route to DLQ
    7. 4. Configuring DLQ in Streams
    8. Example: Branch invalid records
    9. 5. Using DLQ in Connect
    10. Example: Connect DLQ config
    11. 6. Adding Error Metadata
    12. Example: Add error headers
    13. 7. Monitoring DLQ
    14. Example: Watch DLQ lag
    15. 8. Reprocessing DLQ Messages
    16. Example: Replay DLQ
    17. 9. Setting DLQ Retention
    18. Example: 14-day DLQ retention
    19. 10. Implementing Circuit Breaker
    20. Example: Pause on breaker open

  41. Handling Errors and Exceptions

    1. 1. Understanding Producer Exceptions
    2. Example: Callback handling
    3. 2. Handling Timeout Exceptions
    4. Example: Catch timeout
    5. 3. Dealing with Serialization Errors
    6. Example: Guard serialization
    7. 4. Handling Authorization Failures
    8. Example: Handle auth failure
    9. 5. Managing Retriable Exceptions
    10. Example: Safe retries
    11. 6. Handling Non-Retriable Errors
    12. Example: Branch on retriability
    13. 7. Understanding Consumer Exceptions
    14. Example: Wakeup shutdown
    15. 8. Handling Offset Out of Range
    16. Example: Reset policy
    17. 9. Dealing with Rebalance Exceptions
    18. Example: Commit before revoke
    19. 10. Configuring Error Handlers
    20. Example: Streams uncaught handler

  42. Optimizing Producer Performance

    1. 1. Tuning Batch Size
    2. Example: Larger batch
    3. 2. Configuring Linger Time
    4. Example: Linger for batching
    5. 3. Optimizing Compression
    6. Example: Enable compression
    7. 4. Setting Buffer Memory
    8. Example: Buffer memory
    9. 5. Configuring Max In-Flight Requests
    10. Example: In-flight requests
    11. 6. Using Idempotent Producer
    12. Example: Idempotence
    13. 7. Optimizing Serialization
    14. Example: Compact serializer
    15. 8. Tuning Request Timeout
    16. Example: Request timeout
    17. 9. Configuring Retries
    18. Example: Durable producer
    19. 10. Monitoring Producer Metrics
    20. Example: Producer JMX

  43. Optimizing Consumer Performance

    1. 1. Tuning Fetch Size
    2. Example: Larger fetch
    3. 2. Configuring Fetch Wait Time
    4. Example: Fetch wait
    5. 3. Setting Max Poll Records
    6. Example: Poll records
    7. 4. Optimizing Partition Assignment
    8. Example: Assignment strategy
    9. 5. Using Parallel Processing
    10. Example: Worker pool
    11. 6. Configuring Prefetching
    12. Example: Receive buffer
    13. 7. Optimizing Deserialization
    14. Example: Cache size for registry
    15. 8. Tuning Commit Frequency
    16. Example: Commit interval
    17. 9. Using Asynchronous Commits
    18. Example: Async + sync hybrid
    19. 10. Monitoring Consumer Metrics
    20. Example: Consumer lag JMX

  44. Optimizing Broker Performance

    1. 1. Tuning Number of Partitions
    2. Example: Partition count
    3. 2. Optimizing Replica Fetcher Threads
    4. Example: Replica fetchers
    5. 3. Configuring Network Threads
    6. Example: Network threads
    7. 4. Tuning I/O Threads
    8. Example: IO threads
    9. 5. Optimizing Log Segment Size
    10. Example: Segment size
    11. 6. Configuring Zero-Copy Transfer
    12. Example: Zero-copy note
    13. 7. Tuning Flush Settings
    14. Example: Default flush (OS-managed)
    15. 8. Optimizing Page Cache
    16. Example: JVM heap sizing
    17. 9. Setting Socket Buffer Sizes
    18. Example: Socket buffers
    19. 10. Monitoring Broker Metrics
    20. Example: Broker JMX

  45. Optimizing Consumer Rebalancing

    1. 1. Understanding Rebalance Protocol
    2. Example: Cooperative assignor
    3. 2. Configuring Session Timeout
    4. Example: Session timeout
    5. 3. Setting Heartbeat Interval
    6. Example: Heartbeat interval
    7. 4. Configuring Max Poll Interval
    8. Example: Max poll interval
    9. 5. Using Cooperative Rebalancing
    10. Example: Enable cooperative
    11. 6. Implementing Static Membership
    12. Example: Static membership
    13. 7. Minimizing Rebalance Overhead
    14. Example: Graceful leave
    15. 8. Using Incremental Rebalancing
    16. Example: Partitions lost callback
    17. 9. Handling Rebalance Listeners
    18. Example: Subscribe with listener
    19. 10. Monitoring Rebalance Latency
    20. Example: Rebalance JMX

  46. Monitoring Broker Metrics

    1. 1. Understanding JMX Metrics
    2. Example: Enable JMX
    3. 2. Monitoring Bytes In/Out Rate
    4. Example: Bytes in
    5. 3. Tracking Message Rate
    6. Example: Messages in
    7. 4. Monitoring Request Metrics
    8. Example: Request latency
    9. 5. Checking Under-Replicated Partitions
    10. Example: URP metric
    11. 6. Monitoring ISR Shrink/Expand
    12. Example: ISR shrink
    13. 7. Tracking Leader Election Rate
    14. Example: Unclean elections
    15. 8. Monitoring Log Flush Latency
    16. Example: Flush latency
    17. 9. Checking Network Processor Idle
    18. Example: Network idle
    19. 10. Monitoring Request Queue Size
    20. Example: Request queue

  47. Monitoring Producer Metrics

    1. 1. Tracking Record Send Rate
    2. Example: Send rate
    3. 2. Monitoring Batch Size
    4. Example: Batch size avg
    5. 3. Checking Compression Ratio
    6. Example: Compression ratio
    7. 4. Monitoring Record Queue Time
    8. Example: Queue time
    9. 5. Tracking Request Latency
    10. Example: Request latency
    11. 6. Monitoring Buffer Available Bytes
    12. Example: Buffer available
    13. 7. Checking Waiting Threads
    14. Example: Waiting threads
    15. 8. Tracking Record Error Rate
    16. Example: Error rate
    17. 9. Monitoring Record Retry Rate
    18. Example: Retry rate
    19. 10. Checking Metadata Age
    20. Example: Metadata age

  48. Monitoring Consumer Metrics

    1. 1. Tracking Fetch Rate
    2. Example: Fetch rate
    3. 2. Monitoring Records Consumed
    4. Example: Records consumed
    5. 3. Checking Consumer Lag
    6. Example: Lag max
    7. 4. Monitoring Fetch Latency
    8. Example: Fetch latency
    9. 5. Tracking Commit Rate
    10. Example: Commit rate
    11. 6. Monitoring Join Rate
    12. Example: Join rate
    13. 7. Checking Sync Rate
    14. Example: Sync rate
    15. 8. Tracking Heartbeat Response Time
    16. Example: Heartbeat time
    17. 9. Monitoring Last Poll Time
    18. Example: Poll interval
    19. 10. Checking Assigned Partitions
    20. Example: Assigned partitions

  49. Managing Cluster Operations

    1. 1. Adding Brokers to Cluster
    2. Example: New broker config
    3. 2. Removing Brokers from Cluster
    4. Example: Reassign off broker 4
    5. 3. Expanding Cluster
    6. Example: Throttled reassignment
    7. 4. Balancing Partition Load
    8. Example: Generate balance plan
    9. 5. Rolling Restart
    10. Example: Wait for URP=0
    11. 6. Upgrading Kafka Version
    12. Example: Pin protocol version
    13. 7. Migrating to KRaft Mode
    14. Example: Migration flag
    15. 8. Handling Broker Failures
    16. Example: Check leadership after failure
    17. 9. Managing Disk Space
    18. Example: Shrink retention
    19. 10. Configuring Rack Awareness
    20. Example: Set broker rack

  50. Using MirrorMaker for Replication

    1. 1. Understanding MirrorMaker Architecture
    2. Example: Run MM2
    3. 2. Configuring MirrorMaker 2.0
    4. Example: mm2.properties
    5. 3. Setting Source Cluster
    6. Example: Source security
    7. 4. Setting Target Cluster
    8. Example: Target RF
    9. 5. Configuring Topic Replication
    10. Example: Topic selection
    11. 6. Setting Replication Flow
    12. Example: Bidirectional flow
    13. 7. Configuring Consumer Groups Sync
    14. Example: Group offset sync
    15. 8. Setting ACLs Replication
    16. Example: ACL sync
    17. 9. Configuring Heartbeat Connectors
    18. Example: Enable heartbeats
    19. 10. Monitoring Replication Lag
    20. Example: MM2 latency metric

  51. Implementing Multi-Datacenter Replication

    1. 1. Understanding Multi-DC Patterns
    2. Example: Stretch cluster racks
    3. 2. Using Cluster Linking
    4. Example: Create cluster link
    5. 3. Configuring Geo-Replication
    6. Example: WAN compression
    7. 4. Setting Up Disaster Recovery
    8. Example: Translate offsets on DR
    9. 5. Implementing Failover Strategy
    10. Example: Seek on failover
    11. 6. Configuring Conflict Resolution
    12. Example: Region-scoped key
    13. 7. Monitoring Cross-Cluster Lag
    14. Example: Lag from heartbeats
    15. 8. Using Replicator
    16. Example: Replicator connector
    17. 9. Configuring Network Latency Handling
    18. Example: WAN-tuned producer
    19. 10. Testing Failover Procedures
    20. Example: Verify DR consumer position

  52. Deploying Kafka on Cloud Platforms

    1. 1. Using Amazon MSK
    2. Example: MSK IAM client
    3. 2. Deploying on Confluent Cloud
    4. Example: Confluent Cloud client
    5. 3. Using Azure Event Hubs
    6. Example: Event Hubs client
    7. 4. Deploying on Google Cloud
    8. Example: Strimzi Kafka CR
    9. 5. Configuring Cloud-Specific Settings
    10. Example: Private bootstrap
    11. 6. Managing Cloud Kafka Security
    12. Example: Managed Service for Kafka auth (GCP)
    13. 7. Scaling Cloud Kafka Clusters
    14. Example: MSK scale CLI
    15. 8. Monitoring Cloud Deployments
    16. Example: MSK open monitoring scrape
    17. 9. Optimizing Cloud Costs
    18. Example: Enable tiered storage
    19. 10. Implementing Hybrid Cloud Setup
    20. Example: On-prem to cloud mirror

  53. Capacity Planning and Sizing

    1. 1. Estimating Message Throughput
    2. Example: Throughput math
    3. 2. Calculating Partition Count
    4. Example: Partition formula
    5. 3. Sizing Broker Hardware
    6. Example: Heap vs page cache
    7. 4. Estimating Storage Requirements
    8. Example: Storage estimate
    9. 5. Planning Network Bandwidth
    10. Example: NIC sizing
    11. 6. Determining Replication Factor
    12. Example: Durable topic
    13. 7. Sizing Consumer Groups
    14. Example: Group sizing
    15. 8. Planning for Peak Load
    16. Example: Peak headroom plan
    17. 9. Configuring Retention Policies
    18. Example: 7-day retention
    19. 10. Monitoring Resource Utilization
    20. Example: Disk usage check

  54. Testing Kafka Applications

    1. 1. Using EmbeddedKafka
    2. Example: EmbeddedKafka
    3. 2. Creating Test Producer
    4. Example: Test producer
    5. 3. Creating Test Consumer
    6. Example: Test consumer
    7. 4. Testing Streams Topology
    8. Example: TopologyTestDriver
    9. 5. Mocking External Dependencies
    10. Example: MockProducer
    11. 6. Testing Serialization
    12. Example: Serde round-trip
    13. 7. Testing Error Scenarios
    14. Example: Inject send error
    15. 8. Integration Testing
    16. Example: KafkaContainer
    17. 9. Performance Testing
    18. Example: Producer perf test
    19. 10. Load Testing
    20. Example: Throttled soak test

  55. Debugging and Troubleshooting

    1. 1. Enabling Debug Logging
    2. Example: Dynamic log level
    3. 2. Using Broker Logs
    4. Example: Tail server log
    5. 3. Checking Controller Logs
    6. Example: Controller log grep
    7. 4. Analyzing State Change Logs
    8. Example: State change for topic
    9. 5. Using Kafka Dump Log Tool
    10. Example: Dump segment
    11. 6. Checking Topic Metadata
    12. Example: Describe topic
    13. 7. Analyzing Consumer Lag
    14. Example: Describe group lag
    15. 8. Using JMX Console
    16. Example: Enable JMX
    17. 9. Tracing Request Path
    18. Example: Request log timing fields
    19. 10. Debugging Rebalance Issues
    20. Example: Stabilize rebalances

  56. Implementing Backup and Recovery

    1. 1. Understanding Backup Strategies
    2. Example: Why RF isn't backup
    3. 2. Backing Up Topic Data
    4. Example: S3 sink backup
    5. 3. Backing Up Consumer Offsets
    6. Example: Export offsets
    7. 4. Backing Up Schemas
    8. Example: Export all schemas
    9. 5. Backing Up Configuration
    10. Example: Backup topic configs
    11. 6. Implementing Point-in-Time Recovery
    12. Example: Reset group to timestamp
    13. 7. Restoring from Backup
    14. Example: S3 source restore
    15. 8. Recovering Consumer Groups
    16. Example: Restore offsets from file
    17. 9. Testing Recovery Procedures
    18. Example: Verify restored count
    19. 10. Automating Backup Process
    20. Example: Cron config backup