If your rails server or consumer cannot connect to Kafka locally you can try using the command line utilities included with kafka to diagnose whether your kafka is set up properly:
Create a topic:
⚡ kafka-topics --create --partitions 1 --replication-factor 1 --topic new-test-topic --bootstrap-server 127.0.0.1:9092
Created topic new-test-topic.
Describe the created topic:
⚡ kafka-topics --describe --topic new-test-topic --bootstrap-server 127.0.0.1:9092
Topic: new-test-topic TopicId: BnpzD0HXTXGtidmVxGbNuw PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: new-test-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Start writing events to the topic:
⚡ kafka-console-producer --topic new-test-topic --bootstrap-server 127.0.0.1:9092
>
(then write some messages and press enter between each):
> i-typed-this-message-1
>i-typed-this-message-2
(press Ctrl-C when you are done)
Then start a consumer to read the messages you created:
⚡ kafka-console-consumer --topic new-test-topic --from-beginning --bootstrap-server 127.0.0.1:9092
i-typed-this-message-1
i-typed-this-message-2
(press CTRL-C to exit the consumer process)
If all of these work then check that your config/delivery_boy.yml, config/racecar.yml, .env are configured correctly with each other.
- If you installed with brew, you might need to prefix the command:
$(brew --prefix kafka)/bin/kafka-topics --create..., or add$(brew --prefix kafka)/binto your path. - If you installed in some other way you will need to run
bin/kafka-topics ...from the kafka installation directory, or add it to your path.
Check if there is anything running on the port you expect (e.g. 9092):
lsof -i :9092
If there is no output, either the server is not running, or it is listening to a different port.
Check if the server is running, e.g. using ps -ef | grep kafka.
If you installed kafka using brew and followed the post install instructions to run brew services start kafka:
(assuming you set kafka up to run as a service using brew services start kafka):
brew services info kafka
example output:
kafka (homebrew.mxcl.kafka)
Running: ✔
Loaded: ✔
Schedulable: ✘
User: frederickmeissner
PID: 1664
or
brew services list
example output:
Name Status User File
dbus none
kafka started frederickmeissner ~/Library/LaunchAgents/homebrew.mxcl.kafka.plist
[email protected] started frederickmeissner ~/Library/LaunchAgents/[email protected]
[email protected] started frederickmeissner ~/Library/LaunchAgents/[email protected]
postgresql started frederickmeissner ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
postgresql@13 none
redis started frederickmeissner ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
unbound none
zookeeper started frederickmeissner ~/Library/LaunchAgents/homebrew.mxcl.zookeeper.plist
(you should see kafka listed if you started it using brew services start kafka)
Brew may have started the service, but something has gone wrong. You might need more info from the logs
To find the log file examine the service file, e.g.:
cat $(brew --prefix)/Cellar/kafka/3.0.0/homebrew.kafka.service
example output:
[Unit]
Description=Homebrew generated unit for kafka
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
ExecStart=/opt/homebrew/opt/kafka/bin/kafka-server-start /opt/homebrew/etc/kafka/server.properties
Restart=always
WorkingDirectory=/opt/homebrew
StandardOutput=append:/opt/homebrew/var/log/kafka/kafka_output.log
StandardError=append:/opt/homebrew/var/log/kafka/kafka_output.log%
View the logs, e.g. less $(brew --prefix)/var/log/kafka/kafka_output.log for more info.