Apache Kafka. Set up. Writing to. Reading from. Monitoring. Part 4

Now that we have Kafka cluster up and running (Part 1 Part 2), and we are able to monitor it (Part 3), we need to learn how to write to and read from it by using Java.

I have created and uploaded my projects onto github so feel free to download the code and follow along. It is configured to work with IP addresses and topics that we created and configured in Part 1 and 2.

Producer code
First you need to specify connection information. This is as simple as specifying broker list IPs and ports.
// List of brokers that the producer will try to connect to
props.put("metadata.broker.list", "192.168.33.21:9092,192.168.33.22:9092");

Inside of properties you can set number of different choices and flags. Later you would use it to create ProducerConfig object that would in turn be used to create Producer object that would be able to send out messages to Kafka cluster.

Then select a topic that you want to write to
String topic = "my_test_topic" ;

Construct a message that would contain your topic, message and id and use Producer object to send it.

String msg = "Test message # " + 1 ;
KeyedMessage<String, String> data = new KeyedMessage<String, String>(topic, String.valueOf(1), msg);
producer.send(data);

Consumer code
This code is a little bit more complex and I had to use lots of tricks to make sure that I will be able to read from a topic of interest. Basically, if you were not careful with offsets, you would be able to read anything... I tried to comment the code as much as possible so that it would be self explanatory, but the best way to understand it all is by running it with debug and see the process flow.

Hope you enjoyed this start to end Kafka cluster guide and don't hesitate to reach out to me with comments or questions!