I got a bug of kafka on windows. It caused disk space usage issue because kafka borkers can not delete old log files after the configured time.
So, I have to build binary of kafka with patch from Microsoft/kafka’s version to avoid more disk space usage.
I installed kafka(0.10.2.1) on windows2016 not long ago and everything works fine except delete old log files.
P.S:I installed zookeeper on windows2016 too.
This bug only for windows family. so You will get the same issue with me as below:
I found a ticket of kafka on the Apach jira. The ticket number is KAFKA-1194.
I am so surprise for that because this is a bug around four years ago.
Okay complain done, let’s jump to how to build binary from source.
You can skip below content if you did not install kafka/kafka cluster on windows.
Download Microsoft/kafka
P.S:
I have to use this version of kafka because I got a bug of kafka on windows for delete old log.
Note:
Kafka requires Gradle 2.0 or higher.
Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.
First bootstrap and download the wrapper
cd kafka_source_dir
gradle
Building a jar and running it
gradlew jar
Building source jar
gradlew srcJar
Building aggregated Javadoc
gradlew aggregatedJavadoc
Building javadoc and scaladoc
gradlew javadoc
gradlew javadocJar # builds a javadoc jar for each module
gradlew scaladoc
gradlew scaladocJar # builds a scaladoc jar for each module
gradlew docsJar # builds both (if applicable) javadoc and scaladoc jars for each modul
Running unit tests
gradlew test
Building a binary release gzipped tar ball
gradlew clean
gradlew releaseTarG
note:
you will find binary of kafka in .\kafka_source\core\build\distributions after builded work complete.
Please unzip this file then install kafka on your windows server.
I did not get any issue of kafka for delete old log files on windows2016 after I installed this version.
Reference