Syntax highlighter header

Wednesday 24 July 2019

LEADER_NOT_AVAILABLE Kafka error

Recently I installed kafka on an Amazon AWS EC2 instance and tried to post messages to it using my windows machine using public IP address of EC2 instance. But it failed with following error.

23:06:01.622 [kafka-producer-network-thread | producer-1] DEBUG 
org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] 
Sending metadata request (type=MetadataRequest, topics=kafka_core)
 to node 54.165.214.180:9092 (id: -1 rack: null)
23:06:01.956 [kafka-producer-network-thread | producer-1] WARN 
org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1]
 Error while fetching metadata with correlation id 70 : 
{kafka_core=LEADER_NOT_AVAILABLE}

I had modified the following line in server.properties configuration file for kafka to use public IP address of the EC2 instance.

advertised.listeners=PLAINTEXT://54.165.214.180:9092

But it was still not working. I searched the internet and everyone was saying that advertised.listeners needs to be same as host IP address. But for me it was not an option because in EC2 instances public and private IP addresses are different. I needed to connect from outside so advertised.listeners had to be public IP address. I struggled for a week and yesterday I found that if I use hostname instead of public IP address it works.

advertised.listeners=PLAINTEXT://ec2-54-165-214-180.compute-1.amazonaws.com:9092

The above line in server.properties config file for Kafka made it work.