What is Kafka

Real time Data Processing and Storage

Setup

• Download the kafka binary file
• Generate a random uuid and save it anywhere you can remember to source it in the session that kafka server will be running in (e.g. .envrc for Direnv)

echo KAFKA_CLUSTER_ID=$(./bin/kafka-storage.sh random-uuid) >> .envrc

• Change the logs.dir (in config/kraft/server.properties) to a persistent storage directory
• Format the cluster directory

./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

• Start the server

./bin/kafka-server-start.sh config/kraft/server.properties

Tricks

Reset a consumer offset

• Stop the consumer
• Execute the following command

./bin/kafka-consumer-groups.sh kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group group-name --topic topic-name --reset-offsets --to-offset 0 --execute