Skip to content

Commit 4c44abe

Browse files
nIvanovpnIvanovp
committed
init commit
0 parents  commit 4c44abe

19 files changed

+583
-0
lines changed

‎Dockerfile

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FROM java:8
2+
3+
RUN apt-get update
4+
5+
RUN apt-get install -y maven
6+
7+
RUN ls -l
8+
9+
WORKDIR /code
10+
11+
ADD pom.xml /code/pom.xml
12+
13+
ADD src /code/src
14+
15+
CMD ["mvn", "test"]

‎README.md

+93
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
## Pre-Requisites to run this example locally
2+
3+
- install docker-compose [https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/)
4+
- modify the ```KAFKA_ADVERTISED_HOST_NAME``` in ```docker-compose.yml``` to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)
5+
- if you want to customize any Kafka parameters, simply add them as environment variables in ```docker-compose.yml```, e.g. in order to increase the ```message.max.bytes``` parameter set the environment to ```KAFKA_MESSAGE_MAX_BYTES: 2000000```. To turn off automatic topic creation set ```KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'```
6+
7+
## Usage this example locally
8+
9+
Start a cluster:
10+
11+
- ```docker-compose up -d ```
12+
13+
Add more brokers:
14+
15+
- ```docker-compose scale kafka=3```
16+
17+
Destroy a cluster:
18+
19+
- ```docker-compose stop```
20+
21+
## Note
22+
23+
The default ```docker-compose.yml``` should be seen as a starting point. By default each broker will get a new port number and broker id on restart. Depending on your use case this might not be desirable. If you need to use specific ports and broker ids, modify the docker-compose configuration accordingly, e.g. [docker-compose-single-broker.yml](https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-single-broker.yml):
24+
25+
- ```docker-compose -f docker-compose-single-broker.yml up```
26+
27+
## [Instruction : how to build this example in Codefresh](https://docs.codefresh.io/docs/spring-boot-kafka-zookeeper)
28+
29+
- Fork this repo
30+
- Login to Codefresh using your GitHub account
31+
- Go to the tab Composition
32+
- Add a new composition with file from repository
33+
- Choose the path to the docker-compose.yml
34+
- Continue to follow the instructions in the dialog
35+
- Build the images
36+
37+
Now you have a added repository and composition in Codefresh. You can just launch the created composition to see the results.
38+
If you need to change the `kafka` Docker image you can do it in the `./kafka/Dockerfile`.
39+
Also, using the Environment variables you are able to set up the `kafka` service in your composition.
40+
41+
## Broker IDs
42+
43+
You can configure the broker id in different ways
44+
45+
1. explicitly, using ```KAFKA_BROKER_ID```
46+
2. via a command, using ```BROKER_ID_COMMAND```, e.g. ```BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $2}'"```
47+
48+
If you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070). This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.
49+
50+
51+
## Automatically create topics
52+
53+
If you want to have kafka-docker automatically create topics in Kafka during
54+
creation, a ```KAFKA_CREATE_TOPICS``` environment variable can be
55+
added in ```docker-compose.yml```.
56+
57+
Here is an example snippet from ```docker-compose.yml```:
58+
59+
environment:
60+
KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"
61+
62+
```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition, 1 replica and a `cleanup.policy` set to `compact`.
63+
64+
## Advertised hostname
65+
66+
You can configure the advertised hostname in different ways
67+
68+
1. explicitly, using ```KAFKA_ADVERTISED_HOST_NAME```
69+
2. via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"```
70+
71+
When using commands, make sure you review the "Variable Substitution" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/)
72+
73+
If ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes precedence over ```HOSTNAME_COMMAND```
74+
75+
For AWS deployment, you can use the Metadata service to get the container host's IP:
76+
```
77+
HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4
78+
```
79+
Reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
80+
81+
## JMX
82+
83+
For monitoring purposes you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect
84+
85+
* java.rmi.server.hostname - interface to bind listening port
86+
* com.sun.management.jmxremote.rmi.port - The port to service RMI requests
87+
88+
For example, to connect to a kafka running locally (assumes exposing port 1099)
89+
90+
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
91+
JMX_PORT: 1099
92+
93+
Jconsole can now connect at ```jconsole 192.168.99.100:1099```

‎docker-compose.yml

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
version: '2'
2+
services:
3+
zookeeper:
4+
image: wurstmeister/zookeeper
5+
ports:
6+
- "2181:2181"
7+
kafka:
8+
build:
9+
context: kafka
10+
dockerfile: Dockerfile
11+
links:
12+
- zookeeper:zk
13+
ports:
14+
- "9092:9092"
15+
environment:
16+
KAFKA_ADVERTISED_HOST_NAME: $CF_HOST_IP
17+
KAFKA_ZOOKEEPER_CONNECT: zk:2181
18+
KAFKA_MESSAGE_MAX_BYTES: 2000000
19+
KAFKA_CREATE_TOPICS: "Topic1:1:1"
20+
volumes:
21+
- /var/run/docker.sock:/var/run/docker.sock
22+
depends_on:
23+
- zookeeper
24+
25+
spring_boot:
26+
build: .
27+
depends_on:
28+
- kafka
29+
- zookeeper

‎kafka/Dockerfile

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
FROM anapsix/alpine-java
2+
3+
ARG kafka_version=0.10.2.0
4+
ARG scala_version=2.11
5+
6+
MAINTAINER wurstmeister
7+
8+
RUN apk add --update unzip wget curl docker jq coreutils
9+
10+
ENV KAFKA_VERSION=$kafka_version SCALA_VERSION=$scala_version
11+
ADD download-kafka.sh /tmp/download-kafka.sh
12+
RUN chmod a+x /tmp/download-kafka.sh && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka
13+
14+
VOLUME ["/kafka"]
15+
16+
ENV KAFKA_HOME /opt/kafka
17+
ENV PATH ${PATH}:${KAFKA_HOME}/bin
18+
ADD start-kafka.sh /usr/bin/start-kafka.sh
19+
ADD broker-list.sh /usr/bin/broker-list.sh
20+
ADD create-topics.sh /usr/bin/create-topics.sh
21+
# The scripts need to have executable permission
22+
RUN chmod a+x /usr/bin/start-kafka.sh && \
23+
chmod a+x /usr/bin/broker-list.sh && \
24+
chmod a+x /usr/bin/create-topics.sh
25+
# Use "exec" form so that it runs as PID 1 (useful for graceful shutdown)
26+
CMD ["start-kafka.sh"]

‎kafka/README.md

+93
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
## Pre-Requisites to run this example locally
2+
3+
- install docker-compose [https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/)
4+
- modify the ```KAFKA_ADVERTISED_HOST_NAME``` in ```docker-compose.yml``` to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)
5+
- if you want to customize any Kafka parameters, simply add them as environment variables in ```docker-compose.yml```, e.g. in order to increase the ```message.max.bytes``` parameter set the environment to ```KAFKA_MESSAGE_MAX_BYTES: 2000000```. To turn off automatic topic creation set ```KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'```
6+
7+
## Usage this example locally
8+
9+
Start a cluster:
10+
11+
- ```docker-compose up -d ```
12+
13+
Add more brokers:
14+
15+
- ```docker-compose scale kafka=3```
16+
17+
Destroy a cluster:
18+
19+
- ```docker-compose stop```
20+
21+
## Note
22+
23+
The default ```docker-compose.yml``` should be seen as a starting point. By default each broker will get a new port number and broker id on restart. Depending on your use case this might not be desirable. If you need to use specific ports and broker ids, modify the docker-compose configuration accordingly, e.g. [docker-compose-single-broker.yml](https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-single-broker.yml):
24+
25+
- ```docker-compose -f docker-compose-single-broker.yml up```
26+
27+
## Instruction : how to build this example in Codefresh
28+
29+
- Fork this repo
30+
- Login to Codefresh using your GitHub account
31+
- Go to the tab Composition
32+
- Add a new composition with file from repository
33+
- Choose the path to the docker-compose.yml
34+
- Continue to follow the instructions in the dialog
35+
- Build the images
36+
37+
Now you have a added repository and composition in Codefresh. You can just launch the created composition to see the results.
38+
If you need to change the `kafka` Docker image you can do it in the `./kafka/Dockerfile`.
39+
Also, using the Environment variables you are able to set up the `kafka` service in your composition.
40+
41+
## Broker IDs
42+
43+
You can configure the broker id in different ways
44+
45+
1. explicitly, using ```KAFKA_BROKER_ID```
46+
2. via a command, using ```BROKER_ID_COMMAND```, e.g. ```BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $2}'"```
47+
48+
If you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070). This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.
49+
50+
51+
## Automatically create topics
52+
53+
If you want to have kafka-docker automatically create topics in Kafka during
54+
creation, a ```KAFKA_CREATE_TOPICS``` environment variable can be
55+
added in ```docker-compose.yml```.
56+
57+
Here is an example snippet from ```docker-compose.yml```:
58+
59+
environment:
60+
KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"
61+
62+
```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition, 1 replica and a `cleanup.policy` set to `compact`.
63+
64+
## Advertised hostname
65+
66+
You can configure the advertised hostname in different ways
67+
68+
1. explicitly, using ```KAFKA_ADVERTISED_HOST_NAME```
69+
2. via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"```
70+
71+
When using commands, make sure you review the "Variable Substitution" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/)
72+
73+
If ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes precedence over ```HOSTNAME_COMMAND```
74+
75+
For AWS deployment, you can use the Metadata service to get the container host's IP:
76+
```
77+
HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4
78+
```
79+
Reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
80+
81+
## JMX
82+
83+
For monitoring purposes you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect
84+
85+
* java.rmi.server.hostname - interface to bind listening port
86+
* com.sun.management.jmxremote.rmi.port - The port to service RMI requests
87+
88+
For example, to connect to a kafka running locally (assumes exposing port 1099)
89+
90+
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
91+
JMX_PORT: 1099
92+
93+
Jconsole can now connect at ```jconsole 192.168.99.100:1099```

‎kafka/broker-list.sh

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
3+
CONTAINERS=$(docker ps | grep 9092 | awk '{print $1}')
4+
BROKERS=$(for CONTAINER in $CONTAINERS; do docker port $CONTAINER 9092 | sed -e "s/0.0.0.0:/$HOST_IP:/g"; done)
5+
echo $BROKERS | sed -e 's/ /,/g'

‎kafka/create-topics.sh

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/bin/bash
2+
3+
4+
if [[ -z "$START_TIMEOUT" ]]; then
5+
START_TIMEOUT=600
6+
fi
7+
8+
start_timeout_exceeded=false
9+
count=0
10+
step=10
11+
while netstat -lnt | awk '$4 ~ /:'$KAFKA_PORT'$/ {exit 1}'; do
12+
echo "waiting for kafka to be ready"
13+
sleep $step;
14+
count=$(expr $count + $step)
15+
if [ $count -gt $START_TIMEOUT ]; then
16+
start_timeout_exceeded=true
17+
break
18+
fi
19+
done
20+
21+
if $start_timeout_exceeded; then
22+
echo "Not able to auto-create topic (waited for $START_TIMEOUT sec)"
23+
exit 1
24+
fi
25+
26+
if [[ -n $KAFKA_CREATE_TOPICS ]]; then
27+
IFS=','; for topicToCreate in $KAFKA_CREATE_TOPICS; do
28+
echo "creating topics: $topicToCreate"
29+
IFS=':' read -a topicConfig <<< "$topicToCreate"
30+
if [ ${topicConfig[3]} ]; then
31+
JMX_PORT='' $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper $KAFKA_ZOOKEEPER_CONNECT --replication-factor ${topicConfig[2]} --partition ${topicConfig[1]} --topic "${topicConfig[0]}" --config cleanup.policy="${topicConfig[3]}"
32+
else
33+
JMX_PORT='' $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper $KAFKA_ZOOKEEPER_CONNECT --replication-factor ${topicConfig[2]} --partition ${topicConfig[1]} --topic "${topicConfig[0]}"
34+
fi
35+
done
36+
fi

‎kafka/docker-compose-single-broker.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
version: '2'
2+
services:
3+
zookeeper:
4+
image: wurstmeister/zookeeper
5+
ports:
6+
- "2181:2181"
7+
kafka:
8+
build: .
9+
ports:
10+
- "9092:9092"
11+
environment:
12+
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
13+
KAFKA_CREATE_TOPICS: "test:1:1"
14+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
15+
volumes:
16+
- /var/run/docker.sock:/var/run/docker.sock

‎kafka/docker-compose.yml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
version: '2'
2+
services:
3+
zookeeper:
4+
image: wurstmeister/zookeeper
5+
ports:
6+
- "2181:2181"
7+
kafka:
8+
build: .
9+
ports:
10+
- "9092"
11+
environment:
12+
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
13+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
14+
volumes:
15+
- /var/run/docker.sock:/var/run/docker.sock

‎kafka/download-kafka.sh

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/sh
2+
3+
mirror=$(curl --stderr /dev/null https://www.apache.org/dyn/closer.cgi\?as_json\=1 | jq -r '.preferred')
4+
url="${mirror}kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"
5+
wget -q "${url}" -O "/tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"

‎kafka/start-kafka-shell.sh

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#!/bin/bash
2+
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash

0 commit comments

Comments
 (0)