Skip to main content
Mapagam

Managing Topics

1. Creating Topics

Flag Purpose Example
--create Create a new topic Required action
--partitions Number of partitions 6
--replication-factor Copies per partition 3
--config Per-topic override retention.ms=86400000

Example: Create with configs

kafka-topics.sh --create --topic payments \
  --partitions 12 --replication-factor 3 \
  --config retention.ms=604800000 --config min.insync.replicas=2 \
  --bootstrap-server localhost:9092

2. Listing Topics

Flag Purpose Notes
--list Show all topic names One per line
--exclude-internal Hide __ topics Skip internal
--topic Filter by name pattern Exact match

Example: List topics

kafka-topics.sh --list --bootstrap-server localhost:9092 --exclude-internal

3. Describing Topics

Field Meaning Detail
Leader Broker serving the partition Per partition
Replicas All brokers holding a copy Order = preference
Isr In-sync replica set Election candidates

Example: Describe a topic

kafka-topics.sh --describe --topic payments \
  --bootstrap-server localhost:9092

4. Setting Partition Count

Aspect Rule Detail
Increase Only Partitions can be added, never removed One-way operation
Key Ordering Adding partitions changes key→partition mapping Breaks co-location
Parallelism More partitions = more consumer scaling Match to throughput

Example: Add partitions

kafka-topics.sh --alter --topic payments --partitions 24 \
  --bootstrap-server localhost:9092

5. Setting Replication Factor

Aspect Description Detail
At Creation Set via --replication-factor RF ≤ broker count
After Creation Requires reassignment JSON No direct flag
Durability RF=3 standard for production Tolerates 2 losses

Example: Increase RF via reassignment

{"version":1,"partitions":[
  {"topic":"payments","partition":0,"replicas":[1,2,3]}
]}

6. Altering Topic Configuration

Action Command Notes
Set --add-config key=value Override default
Remove --delete-config key Revert to default
Describe --describe --all Show effective config

Example: Alter retention

kafka-configs.sh --alter --entity-type topics --entity-name payments \
  --add-config retention.ms=259200000 \
  --bootstrap-server localhost:9092

7. Deleting Topics

Aspect Description Detail
--delete Marks topic for deletion Async cleanup
delete.topic.enable Must be true on broker Default true
Irreversible All data and offsets removed No recovery
Warning: Topic deletion is permanent and removes all partition data. Confirm before running in production.

Example: Delete a topic

kafka-topics.sh --delete --topic obsolete-topic \
  --bootstrap-server localhost:9092

8. Configuring Topic Retention

Config Description Default
retention.ms Time before deletion 604800000 (7d)
retention.bytes Max partition size -1 (unlimited)
-1 value Disables that limit Infinite retention

Example: Time + size retention

kafka-configs.sh --alter --entity-type topics --entity-name logs \
  --add-config retention.ms=86400000,retention.bytes=1073741824 \
  --bootstrap-server localhost:9092

9. Setting Cleanup Policy

Policy Behavior Use Case
delete Remove old segments by age/size Event streams
compact Keep latest value per key Changelogs, state
compact,delete Compact then enforce retention Bounded changelog

Example: Enable compaction

kafka-configs.sh --alter --entity-type topics --entity-name user-profiles \
  --add-config cleanup.policy=compact \
  --bootstrap-server localhost:9092

10. Configuring Segment Settings

Config Description Default
segment.bytes Max segment file size 1073741824 (1GB)
segment.ms Force roll after time 604800000 (7d)
segment.index.bytes Max index file size 10485760

Example: Smaller segments for faster compaction

kafka-configs.sh --alter --entity-type topics --entity-name user-profiles \
  --add-config segment.bytes=104857600,segment.ms=3600000 \
  --bootstrap-server localhost:9092

Was this article helpful?

Related Cheat Sheets