Kafka for local development of applications:<\/strong>.<\/p>\n There are multiple ways of running Kafka locally for development of apps but the easiest method is by Docker Compose<\/strong> is included as part of this desktop.docker-compose<\/em>facilitates installing Kafka and Zookeeper with the help of Create a file called docker-compose.yml<\/em> in your project directory and paste the following lines of code: The above compose file defines two services: zookeeper and kafka. The zookeeper service uses a public zookeeper image and the kafka service uses a public kafka image pulled from the Docker Hub registry. <\/p>\n 1. Start the Kafka service<\/strong>. To list all running docker containers, run the following command<\/p>\n You can shut down docker-compose by executing the following command in another terminal.<\/p>\n Start the kafka service and check the ZooKeeper logs to verify that ZooKeeper is working and healthy.<\/p>\n Next, check the Kafka logs to verify that broker is working and healthy.<\/p>\n Two fundamental concepts in Kafka are Topics<\/em> and Partitions<\/em>. Kafka topics are divided into a number of partitions. While the topic is a logical concept in Kafka, a partition is the smallest storage unit that holds a subset of records owned by a topic. Each partition is a single log file where records are written to it in an append-only fashion.<\/p>\n 2. Create a Kafka topic<\/strong> Use docker compose exec<\/em> to execute a command in a running container. For example, docker compose exec<\/em> command by default allocates a TTY, so that you can use such a command to get an interactive prompt. Go into directory where docker-compose.yml<\/em> file present, and execute it as<\/p>\n (for zookeeper $docker compose exec zookeeper bash<\/em>)<\/p>\n Change the directory to \/opt\/kafka\/bin where you find scripts such as kafka-topics.sh<\/em>.<\/p>\n Kafka topics are divided into a number of partitions. Each partition in a topic is an ordered, immutable sequence of records that continually appended.<\/p>\n Create a new topic or, list or delete existing topics:<\/p>\n If necessary, delete a topic using the following command.<\/p>\n 3. How to produce and consume messages from kafka Topic using the command line?<\/strong> Kafka broker (a.k.a Kafka server\/node) is the server node in the cluster, mainly responsible for hosting partitions of Kafka topics, transferring messages from Kafka producer to Kafka consumer and, providing data replication and partitioning within a Kafka Cluster.<\/p>\n The following is a producer command line to read data from standard input and write it to a Kafka topic. The following is a command line to read data from a Kafka topic and write it to standard output. 4. How to consume messages from kafka Topic using java web application?<\/strong><\/p>\n Another way of reading data from a Kafka topic is by simply using a Java Spring Boot<\/em>.<\/p>\n The following demonstrates how to receive messages from Kafka topic. First in this blog I create a Spring Kafka Consumer, which is able to listen the messages sent to a Kafka topic using the commandline. Then, I create a Spring Kafka Producer, which is able to send messages to a Kafka topic.<\/p>\ndocker-compose<\/em><\/code>. First download Docker Desktop,Docker Hub<\/a>and SignIn with Docker ID.<\/ul>\ndocker-compose.yml<\/em><\/code>
\nfile.<\/p>\n
\n![\"\"](\"http:\/\/sys4seq.com\/wp-content\/uploads\/2020\/04\/docker_compose-300x259.png\")
<\/p>\n
\nOpen a terminal, go to the directory where you have the docker-compose.yml<\/em> file, and execute the following command. This command starts the docker-compose engine, and downloads the images and runs them. <\/p>\n$docker compose up -d<\/em> <\/code><\/p><\/blockquote>\n$docker compose ps<\/code> <\/p><\/blockquote>\n$docker compose down<\/code><\/p><\/blockquote>\n$docker compose logs zookeeper | grep -i binding<\/code> <\/p><\/blockquote>\n$docker compose logs kafka | grep -i started<\/code><\/p><\/blockquote>\n
\nThe Kafka cluster stores streams of records in categories called topics. Each record in a topic consists of a key, a value, and a timestamp. A topic can have zero, one, or many consumers that subscribe to the data written to it.<\/p>\n$docker compose exec kafka bash<\/code> <\/p><\/blockquote>\ncd \/opt\/kafka\/bin<\/code><\/p><\/blockquote>\n![\"\"](\"http:\/\/sys4seq.com\/wp-content\/uploads\/2020\/04\/create_kafka_topic-300x160.png\")
<\/p><\/blockquote>\nbash-4.4# .\/kafka-topics.sh --list --bootstrap-server localhost:9092<\/code> <\/p><\/blockquote>\nbash-4.4# .\/kafka-topics.sh --delete --topic mytopic --bootstrap-server localhost:9092<\/code><\/p><\/blockquote>\n
\nKafka producers are those client applications that publish (write) events to Kafka, and Kafka consumers are those that subscribe (read and process) to these events. A producer can use a partition key to direct messages to a specific partition. If a producer doesn\u2019t specify a partition key when producing a record, Kafka will use a round-robin partition assignment.<\/p>\n
\n![\"\"](\"http:\/\/sys4seq.com\/wp-content\/uploads\/2020\/04\/kafka_producer-300x174.png\")
<\/p>\n
\n![\"\"](\"http:\/\/sys4seq.com\/wp-content\/uploads\/2020\/04\/Kafka_consumer-300x171.png\")
<\/p>\n
![\"\"](\"http:\/\/sys4seq.com\/wp-content\/uploads\/2020\/04\/spring_initlizr-300x89.png\")
![\"\"](\"http:\/\/sys4seq.com\/wp-content\/uploads\/2020\/04\/application_yml-300x88.png\")
<\/p>\n