RiCo技術農場

這篇紀錄一下在CentOS 7安裝kafka cluster過程，很多類似步驟可參考Install zookeeper on CentOS 7

@install JDK and configure OS

CentOS預設都有JDK，目錄為/usr/lib/jvm，輸入$ java –version確認

vi設定環境變數/etc/profile(所有使用者皆可使用)並附加下面路徑

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk

export JRE_HOME=/usr/lib/jvm/jre

$ source /etc/profile #立即生效

@install kafka 1.0

$ sudo wget http://apache.stu.edu.tw/kafka/1.0.0/kafka_2.11-1.0.0.tgz
$ sudo mkdir kafka1.0
$ sudo tar -xvzf kafka_2.11-1.0.0.tgz -C kafka1.0  #如目錄錯誤，可透過sudo rm -rf folder整個刪除
$ sudo rm kafka_2.11-1.0.0.tgz

加入KAFKA_HOME的環境變數(/etc/profile) #pwd 顯示目前路徑

export KAFKA_HOME=/opt/kafka1.0/kafka_2.11-1.0.0

source /etc/profile #立即生效

@configure server.properties

$ sudo mkdir /var/kafka/kafka-logs-2

更改差異如下

broker.id=0~2

port=9092~4

log.dirs=/var/kafka/kafka-logs~2

note: create a kafka cluster via different port on the same host

P.S:

I just demo but you have to configure it by yourself in your environment.

Don’t copy it for your production.

 

@create service file

Service type(來源:鳥哥)

下面是我建立kafka的服務腳本

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target 
After=network.target remote-fs.target zookeeper2181.service

[Service]
Type=simple
Environment=JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
ExecStart=/opt/kafka1.0/kafka_2.11-1.0.0/bin/kafka-server-start.sh /opt/kafka1.0/kafka_2.11-1.0.0/config/server.properties
ExecStop=/opt/kafka1.0/kafka_2.11-1.0.0/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

由於我會啟動三個instance，所以我會建立三個.service file

$ sudo vi /etc/systemd/system/kafka9092.service
sudo systemctl start kafka9092.service #啟動服務
sudo systemctl status kafka9092.service #查看服務狀態

9092

9093

9094

要關閉服務可透過 sudo systemctl stop kafka9092.service

都沒問題後，我們設定開機自動執行kafka9092.service

$ sudo systemctl enable kafka9092.service

 

@configure firewall

因為我們client是直接存取kafka cluster，所以我需要開放port 9092~9094

$ sudo firewall-cmd --list-all --permanent #顯示永遠設定
$ sudo firewall-cmd --permanent --add-port=9092/tcp  #永遠開啟kafka 服務port 9092~4
$ sudo firewall-cmd --reload

@create topic

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic test-topic
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-topic

@test producer and consumer via  c#’s client

#consumer
./kafka-console-consumer.sh --bootstrap-server 192.168.137.10:9092 --topic test-topic --from-beginning

#producer
./kafka-console-producer.sh --broker-list 192.168.137.10:9092 --topic test-topic

@C#’s client

#consumer

#producer

 

參考

Kafka 1.0 Documentation

Install zookeeper on CentOS 7