kafka-mirrormaker-azure-tutorial

This tutorial is a simple step by step instruction to reach the following goal:

• Install a on-prem Apache Kafka
• Configure the Apache Kafka tool mirrormaker
• Send message produced on-prem to Microsoft Azure Event Hub

Please note: This tutorial is only for dev / poc (proof of concept) environment and not for production environment.

Step 1

Prerequisite:

• Ubuntu 20.04 (on Windows 10 with WSL 2)
• MS Visual Studio Code
• Microsoft Azure subscription

Step 2

Install java on Ubuntu 20.04:

$ sudo apt update

$ sudo apt install openjdk-8-jdk -y

$ java -version

$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

$ export PATH=$PATH:$JAVA_HOME/bin

Step 3

Install Apache Kafka on Ubuntu 20.04:

$ mkdir kafka

$ cd kafka

$ wget https://downloads.apache.org/kafka/2.7.0/kafka_2.13-2.7.0.tgz

$ tar -zxf kafka_2.13-2.7.0.tgz

$ sudo mv kafka_2.13-2.7.0 /usr/local/kafka

$ sudo mkdir /tmp/kafka-logs

$ sudo mkdir /tmp/zookeeper

Step 4

Start Apache Zookeeper on Ubuntu 20.04:

$ sudo /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties

Step 5

Start Apache Kafka Server on Ubuntu 20.04:

$ sudo /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties

Step 6

Create Kafka topic on Ubuntu 20.04:

$ /usr/local/kafka/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

$ /usr/local/kafka/bin/kafka-topics.sh --create --topic mymachine --bootstrap-server localhost:9092

$ /usr/local/kafka/bin/kafka-topics.sh --list --bootstrap-server  localhost:9092

Step 7

Microsoft Azure Event-Hub:

Go to your Microsoft Azure subscription.

Create a resource

Create

You have to create a "Standard" tier Event Hubs namespace: In this case the Kafka endpoint for the namespace is automatically enabled.

Next: Features >  Next: Tags > Next: Review + create > Create

Go to resource

Click on 

and copy the "Connection string–primary key" 

e.g. Endpoint=sb://kafkaazure.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[your key]

for later use.

Step 8

Configure MirrorMaker on Ubuntu 20.04:

$ cd /usr/local/kafka/config

$ code .

Visual Studio Code opens. Create File / New File and enter the following configuration:

bootstrap.servers=localhost:9092

exclude.internal.topics=true

client.id=mirror_maker_consumer
group.id=mirror_maker_consumer

File / Save as sourceCluster1Consumer.config

Create File / New File and enter the following configuration:

bootstrap.servers=kafkaazure.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://kafkaazure.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[your key];

acks=1

batch.size=50

client.id=mirror_maker_test_producer

File / Save as azureClusterProducer.config

Step 9

Start MirrorMaker on Ubuntu 20.04:

$ sudo /usr/local/kafka/bin/kafka-mirror-maker.sh --consumer.config /usr/local/kafka/config/sourceCluster1Consumer.config --num.streams 1 --producer.config /usr/local/kafka/config/azureClusterProducer.config --whitelist=".*"

Step 10

Start Kafka producer on Ubuntu 20.04:

$ sudo /usr/local/kafka/bin/kafka-console-producer.sh --topic mymachine --bootstrap-server localhost:9092

Enter the following value:

{"sensor_id": 1,"ltime": 1613204245,"temp": 5.5,"status": 1}

Step 11

Start Kafka consumer on Ubuntu 20.04:

$ sudo /usr/local/kafka/bin/kafka-console-consumer.sh --topic mymachine --from-beginning --bootstrap-server localhost:9092

Step 12

Microsoft Azure Event-Hub:

In the Event Hubs Namespace go to Event Hubs:

An Event Hub with the name mymachine ist generated automatically. Click on the mymachine Event Hub:

Go to mymachine: Your first on-prem produced Kafka message has been transmitted to Microsoft Azure as you can see for example in "Messages".

Please note that it can take a short instance for the messages to appear in the dashboard below.

Next Steps

• azure-streaminganalytics-SQLdb-PowerBi-Grafana-tutorial