Understanding Apache Kafka

Apache Kafka is like a giant message board used by computer programs to talk to each other. Imagine a large bulletin board in a common area where different people come to leave notes (messages) for others to read. Kafka acts as this bulletin board in the world of software.

Key Concepts in Simple Terms

Producers: These are like people who write notes and post them on the bulletin board. In Kafka, producers are programs that send messages to Kafka.

Consumers: These are like people who come to read the notes posted on the bulletin board. In Kafka, consumers are programs that read messages from Kafka.

Topics: These are like different sections of the bulletin board dedicated to specific subjects. For instance, one section could be for "Weather Updates" and another for "Sports Scores." In Kafka, topics are categories where messages are posted.

Brokers: These are like the caretakers of the bulletin board, making sure the notes are posted properly and are available for others to read. In Kafka, brokers are servers that store and manage the messages.

What is Kafka Used For?

Kafka is used to handle real-time data feeds. It's great for scenarios where you need to process a large volume of data quickly and reliably. Here are some common uses:

• Real-Time Data Streaming: Companies use Kafka to stream data in real-time from one system to another. For example, financial institutions use it to stream stock prices and trades.
• Activity Tracking: Websites and apps use Kafka to track user activities, such as clicks, searches, or views, in real-time.
• Log Aggregation: Kafka can collect logs from multiple servers and store them centrally, making it easier to monitor and analyze.
• Event Sourcing: It's used to capture changes (events) in the state of an application, such as order statuses in an e-commerce system.
• Data Integration: Kafka can integrate data from different systems, making it a central hub for data movement.

Where Do We Use Kafka?

Kafka is used in various industries and applications, including:

• Technology and Social Media: Companies like LinkedIn and Twitter use Kafka to handle large volumes of user activity data.
• Financial Services: Banks and trading firms use Kafka for real-time analytics and fraud detection.
• E-commerce: Online retailers use Kafka to track orders, manage inventory, and provide real-time recommendations to users.
• Telecommunications: Telecom companies use Kafka to monitor network activity and performance.

Summary

In essence, Apache Kafka is a powerful tool for managing and processing streams of data in real-time. It's like a sophisticated bulletin board system where producers (writers) post messages (data), and consumers (readers) read them, all organized into topics (sections) and managed by brokers (caretakers). It's widely used in various fields to handle large-scale, real-time data flows efficiently and reliably.

How to install Kafka on Windows

Install Kafka on Windows

- Zookeper start command

C:\WorkSpace\kafka_2.12-3.4.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties

- Kafka server start command:

C:\WorkSpace\kafka_2.12-3.4.0>bin\windows\kafka-server-start.bat config\server.properties

- Topic create command

Go to C:\WorkSpace\kafka_2.12-3.4.0\bin\windows> kafka-topics.bat --create --topic user-topic --bootstrap-server localhost:9092

Here user-topic is the name of topic.

On command prompt it will display like this once topic created:

Created topic user-topic.

- Produce new topic

C:\WorkSpace\kafka_2.12-3.4.0\bin\windows>kafka-console-producer.bat --topic user-topic --bootstrap-server localhost:9092
>Hi
>hello
>how
>are
>you
>dear
>kafka

- Consume above topic messages (open new terminal window)

C:\WorkSpace\kafka_2.12-3.4.0\bin\windows>kafka-console-consumer.bat --topic user-topic --from-beginning --bootstrap-server localhost:9092
Hi
hello
how
are
you
dear
kafka

- To consume only the latest message from a Kafka topic, you can use the --max-messages option with a value of 1. This will consume only the latest message from the topic.Here's an example command:

kafka-console-consumer.bat --topic location-update-topic --bootstrap-server localhost:9092 --max-messages 1