Installing - Apache Kafka on macOS Sierra
This is my step by step guide to installing Apache Kafka 0.10.1.0 n macOS Sierra.
Pre-Requisirtes
Hardware
Model Name: MacBook Pro
Model Identifier: MacBookPro12,1
Processor Name: Intel Core i7
Processor Speed: 3.1 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 4 MB
Memory: 16 GB
Software
OS: macOS Sierra - OS X 10.12. - pre-release version
Package Manager: Home Brew - 1.18
Big Data: Hadoop
Cluster Computing Framework: Apache Spark 2.0.0
Stream Processing Platform: Apache Kafka - 0.10.1.0
As described by Wiki. Apache Kafka is an open-source stream processing platform developed by the Apache Software foundation written in Scala and Java.
Installation
I use HomeBrew to install and update Kafka.
For more information on installing HomeBrew or installing Spark visit my previous blogs.
$ brew search kafka
$ brew install kafka
If you don't have required dependencies like zookeeper and just want to do a standalone Kafka installation. then this will also install the required dependencies for you.
Starting Kafka
Start Dependency:
In order to start zookeeper
$ zkserver start
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
Once the dependency is started,
Start Kafka:
Navigate to the kafka bin.
$ cd /usr/local/Cellar/kafka/0.10.0.1/bin/
$ ls
connect-distributed kafka-consumer-offset-checker kafka-replica-verification kafka-verifiable-consumer
connect-standalone kafka-consumer-perf-test kafka-run-class kafka-verifiable-producer
kafka-acls kafka-mirror-maker kafka-server-start zookeeper-security-migration
kafka-configs kafka-preferred-replica-election kafka-server-stop zookeeper-server-start
kafka-console-consumer kafka-producer-perf-test kafka-simple-consumer-shell zookeeper-server-stop
kafka-console-producer kafka-reassign-partitions kafka-streams-application-reset zookeeper-shell
kafka-consumer-groups kafka-replay-log-producer kafka-topics
$ ./kafka-server-start /usr/local/etc/kafka/server.properties
You should see the following when the server starts:
[2016-12-08 11:21:58,174] INFO KafkaConfig values:
advertised.host.name = null
metric.reporters = []
quota.producer.default = 9223372036854775807
offsets.topic.num.partitions = 50
log.flush.interval.messages = 9223372036854775807
auto.create.topics.enable = true
controller.socket.timeout.ms = 30000
log.flush.interval.ms = null
principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
replica.socket.receive.buffer.bytes = 65536
min.insync.replicas = 1
replica.fetch.wait.max.ms = 500
num.recovery.threads.per.data.dir = 1
ssl.keystore.type = JKS
sasl.mechanism.inter.broker.protocol = GSSAPI
default.replication.factor = 1
ssl.truststore.password = null
log.preallocate = false
sasl.kerberos.principal.to.local.rules = [DEFAULT]
fetch.purgatory.purge.interval.requests = 1000
ssl.endpoint.identification.algorithm = null
replica.socket.timeout.ms = 30000
message.max.bytes = 1000012
num.io.threads = 8
offsets.commit.required.acks = -1
log.flush.offset.checkpoint.interval.ms = 60000
delete.topic.enable = false
quota.window.size.seconds = 1
ssl.truststore.type = JKS
offsets.commit.timeout.ms = 5000
quota.window.num = 11
zookeeper.connect = localhost:2181
authorizer.class.name =
num.replica.fetchers = 1
log.retention.ms = null
log.roll.jitter.hours = 0
log.cleaner.enable = true
offsets.load.buffer.size = 5242880
log.cleaner.delete.retention.ms = 86400000
ssl.client.auth = none
controlled.shutdown.max.retries = 3
queued.max.requests = 500
offsets.topic.replication.factor = 3
log.cleaner.threads = 1
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
socket.request.max.bytes = 104857600
ssl.trustmanager.algorithm = PKIX
zookeeper.session.timeout.ms = 6000
log.retention.bytes = -1
log.message.timestamp.type = CreateTime
sasl.kerberos.min.time.before.relogin = 60000
zookeeper.set.acl = false
connections.max.idle.ms = 600000
offsets.retention.minutes = 1440
replica.fetch.backoff.ms = 1000
inter.broker.protocol.version = 0.10.0-IV1
log.retention.hours = 168
num.partitions = 1
broker.id.generation.enable = true
listeners = null
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
log.roll.ms = null
log.flush.scheduler.interval.ms = 9223372036854775807
ssl.cipher.suites = null
log.index.size.max.bytes = 10485760
ssl.keymanager.algorithm = SunX509
security.inter.broker.protocol = PLAINTEXT
replica.fetch.max.bytes = 1048576
advertised.port = null
log.cleaner.dedupe.buffer.size = 134217728
replica.high.watermark.checkpoint.interval.ms = 5000
log.cleaner.io.buffer.size = 524288
sasl.kerberos.ticket.renew.window.factor = 0.8
zookeeper.connection.timeout.ms = 6000
controlled.shutdown.retry.backoff.ms = 5000
log.roll.hours = 168
log.cleanup.policy = delete
host.name =
log.roll.jitter.ms = null
max.connections.per.ip = 2147483647
offsets.topic.segment.bytes = 104857600
background.threads = 10
quota.consumer.default = 9223372036854775807
request.timeout.ms = 30000
log.message.format.version = 0.10.0-IV1
log.index.interval.bytes = 4096
log.dir = /tmp/kafka-logs
log.segment.bytes = 1073741824
log.cleaner.backoff.ms = 15000
offset.metadata.max.bytes = 4096
ssl.truststore.location = null
group.max.session.timeout.ms = 300000
ssl.keystore.password = null
zookeeper.sync.time.ms = 2000
port = 9092
log.retention.minutes = null
log.segment.delete.delay.ms = 60000
log.dirs = /usr/local/var/lib/kafka-logs
controlled.shutdown.enable = true
compression.type = producer
max.connections.per.ip.overrides =
log.message.timestamp.difference.max.ms = 9223372036854775807
sasl.kerberos.kinit.cmd = /usr/bin/kinit
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
auto.leader.rebalance.enable = true
leader.imbalance.check.interval.seconds = 300
log.cleaner.min.cleanable.ratio = 0.5
replica.lag.time.max.ms = 10000
num.network.threads = 3
ssl.key.password = null
reserved.broker.max.id = 1000
metrics.num.samples = 2
socket.send.buffer.bytes = 102400
ssl.protocol = TLS
socket.receive.buffer.bytes = 102400
ssl.keystore.location = null
replica.fetch.min.bytes = 1
broker.rack = null
unclean.leader.election.enable = true
sasl.enabled.mechanisms = [GSSAPI]
group.min.session.timeout.ms = 6000
log.cleaner.io.buffer.load.factor = 0.9
offsets.retention.check.interval.ms = 600000
producer.purgatory.purge.interval.requests = 1000
metrics.sample.window.ms = 30000
broker.id = 0
offsets.topic.compression.codec = 0
log.retention.check.interval.ms = 300000
advertised.listeners = null
leader.imbalance.per.broker.percentage = 10
(kafka.server.KafkaConfig)
[2016-12-08 11:21:58,248] INFO starting (kafka.server.KafkaServer)
[2016-12-08 11:21:58,266] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2016-12-08 11:21:58,280] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-12-08 11:21:58,294] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:host.name=25.211.171.214 (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.version=1.8.0_111 (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.class.path=:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/argparse4j-0.5.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/connect-api-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/connect-file-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/connect-json-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/connect-runtime-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/guava-18.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/hk2-api-2.4.0-b34.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/hk2-locator-2.4.0-b34.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/hk2-utils-2.4.0-b34.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jackson-annotations-2.6.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jackson-core-2.6.3.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jackson-databind-2.6.3.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/javassist-3.18.2-GA.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/javax.annotation-api-1.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/javax.inject-1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/javax.inject-2.4.0-b34.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/javax.ws.rs-api-2.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-client-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-common-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-container-servlet-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-guava-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-media-jaxb-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jersey-server-2.22.2.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-http-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-io-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-security-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-server-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jetty-util-9.2.15.v20160210.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/jopt-simple-4.9.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka-clients-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka-log4j-appender-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka-streams-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka-streams-examples-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka-tools-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka_2.11-0.10.0.1-sources.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka_2.11-0.10.0.1-test-sources.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/kafka_2.11-0.10.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/log4j-1.2.17.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/lz4-1.3.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/metrics-core-2.2.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/reflections-0.9.10.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/rocksdbjni-4.8.0.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/scala-library-2.11.8.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/slf4j-api-1.7.21.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/slf4j-log4j12-1.7.21.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/snappy-java-1.1.2.6.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/validation-api-1.1.0.Final.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/zkclient-0.8.jar:/usr/local/Cellar/kafka/0.10.0.1/libexec/bin/../libs/zookeeper-3.4.6.jar (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.library.path=/Users/************/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.io.tmpdir=/var/folders/lp/knc8kx754b3dc61s0_hz4mym0000gn/T/ (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:java.compiler= (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:os.version=10.12.2 (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:user.name=************ (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,294] INFO Client environment:user.home=/Users/************ (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,295] INFO Client environment:user.dir=/usr/local/Cellar/kafka/0.10.0.1/bin (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,296] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@5223e5ee (org.apache.zookeeper.ZooKeeper)
[2016-12-08 11:21:58,318] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2016-12-08 11:21:58,322] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-12-08 11:21:58,411] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-12-08 11:21:58,567] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x158df3ad5080000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2016-12-08 11:21:58,569] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-12-08 11:21:58,828] INFO Loading logs. (kafka.log.LogManager)
[2016-12-08 11:21:58,904] INFO Completed load of log __consumer_offsets-0 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,913] INFO Completed load of log __consumer_offsets-1 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,918] INFO Completed load of log __consumer_offsets-10 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,921] INFO Completed load of log __consumer_offsets-11 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,925] INFO Completed load of log __consumer_offsets-12 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,929] INFO Completed load of log __consumer_offsets-13 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,933] INFO Completed load of log __consumer_offsets-14 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,937] INFO Completed load of log __consumer_offsets-15 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,944] INFO Completed load of log __consumer_offsets-16 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,951] INFO Completed load of log __consumer_offsets-17 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,963] INFO Completed load of log __consumer_offsets-18 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,970] INFO Completed load of log __consumer_offsets-19 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,974] INFO Completed load of log __consumer_offsets-2 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,979] INFO Completed load of log __consumer_offsets-20 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,986] INFO Completed load of log __consumer_offsets-21 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:58,995] INFO Completed load of log __consumer_offsets-22 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,008] INFO Completed load of log __consumer_offsets-23 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,018] INFO Completed load of log __consumer_offsets-24 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,024] INFO Completed load of log __consumer_offsets-25 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,036] INFO Completed load of log __consumer_offsets-26 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,040] INFO Completed load of log __consumer_offsets-27 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,045] INFO Completed load of log __consumer_offsets-28 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,049] INFO Completed load of log __consumer_offsets-29 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,056] INFO Completed load of log __consumer_offsets-3 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,062] INFO Completed load of log __consumer_offsets-30 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,073] INFO Completed load of log __consumer_offsets-31 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,080] INFO Completed load of log __consumer_offsets-32 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,084] INFO Completed load of log __consumer_offsets-33 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,091] INFO Completed load of log __consumer_offsets-34 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,099] INFO Completed load of log __consumer_offsets-35 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,103] INFO Completed load of log __consumer_offsets-36 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,108] INFO Completed load of log __consumer_offsets-37 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,112] INFO Completed load of log __consumer_offsets-38 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,122] INFO Completed load of log __consumer_offsets-39 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,128] INFO Completed load of log __consumer_offsets-4 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,138] INFO Completed load of log __consumer_offsets-40 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,145] INFO Completed load of log __consumer_offsets-41 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,150] INFO Completed load of log __consumer_offsets-42 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,153] INFO Completed load of log __consumer_offsets-43 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,156] INFO Completed load of log __consumer_offsets-44 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,160] INFO Completed load of log __consumer_offsets-45 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,166] INFO Completed load of log __consumer_offsets-46 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,169] INFO Completed load of log __consumer_offsets-47 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,173] INFO Completed load of log __consumer_offsets-48 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,176] INFO Completed load of log __consumer_offsets-49 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,181] INFO Completed load of log __consumer_offsets-5 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,188] INFO Completed load of log __consumer_offsets-6 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,194] INFO Completed load of log __consumer_offsets-7 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,215] INFO Completed load of log __consumer_offsets-8 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,220] INFO Completed load of log __consumer_offsets-9 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,236] INFO Completed load of log DEV.RDM.TO.RDS.CORE.Organization.Organization-0 with log end offset 7859 (kafka.log.Log)
[2016-12-08 11:21:59,240] INFO Completed load of log DEV.RDM.TO.RDS.CORE.Organization.OrganizationIdentifier-0 with log end offset 7859 (kafka.log.Log)
[2016-12-08 11:21:59,244] INFO Completed load of log DEV.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnit-0 with log end offset 7967 (kafka.log.Log)
[2016-12-08 11:21:59,248] INFO Completed load of log DEV.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnitIdentifier-0 with log end offset 7967 (kafka.log.Log)
[2016-12-08 11:21:59,255] INFO Completed load of log Golden.IMS.CDC.3NF.GLOBAL.RDM.TEST-0 with log end offset 0 (kafka.log.Log)
[2016-12-08 11:21:59,264] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Location.ActivityCenter-0 with log end offset 7968 (kafka.log.Log)
[2016-12-08 11:21:59,276] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Location.ActivityCenterIdentifier-0 with log end offset 7964 (kafka.log.Log)
[2016-12-08 11:21:59,283] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Location.Location-0 with log end offset 7515 (kafka.log.Log)
[2016-12-08 11:21:59,287] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Location.LocationIdentifier-0 with log end offset 7501 (kafka.log.Log)
[2016-12-08 11:21:59,291] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Organization.OperationalActivityCenter-0 with log end offset 7975 (kafka.log.Log)
[2016-12-08 11:21:59,295] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Organization.OperationalActivityCenterIdentifier-0 with log end offset 7967 (kafka.log.Log)
[2016-12-08 11:21:59,302] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Organization.Organization-0 with log end offset 7982 (kafka.log.Log)
[2016-12-08 11:21:59,305] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Organization.OrganizationIdentifier-0 with log end offset 7970 (kafka.log.Log)
[2016-12-08 11:21:59,309] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnit-0 with log end offset 8007 (kafka.log.Log)
[2016-12-08 11:21:59,312] INFO Completed load of log TEST.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnitIdentifier-0 with log end offset 7989 (kafka.log.Log)
[2016-12-08 11:21:59,318] INFO Completed load of log TEST.RDM.TO.RDS.LOCAL.Organization.Outlet-0 with log end offset 7971 (kafka.log.Log)
[2016-12-08 11:21:59,322] INFO Logs loading complete. (kafka.log.LogManager)
[2016-12-08 11:21:59,454] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-12-08 11:21:59,457] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-12-08 11:21:59,514] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2016-12-08 11:21:59,518] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-12-08 11:21:59,551] INFO [ExpirationReaper-0], Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-08 11:21:59,553] INFO [ExpirationReaper-0], Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-08 11:21:59,635] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-08 11:21:59,646] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-08 11:21:59,646] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-08 11:22:00,346] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-12-08 11:22:00,348] INFO [ExpirationReaper-0], Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-08 11:22:00,348] INFO [ExpirationReaper-0], Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-08 11:22:00,370] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2016-12-08 11:22:00,371] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2016-12-08 11:22:00,373] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:00,412] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-08 11:22:00,414] INFO [ThrottledRequestReaper-Fetch], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-08 11:22:00,426] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-12-08 11:22:00,470] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-08 11:22:00,475] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-08 11:22:00,480] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(25.211.171.214,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2016-12-08 11:22:00,505] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-08 11:22:00,505] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-08 11:22:00,506] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-12-08 11:22:00,808] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [__consumer_offsets,32],[__consumer_offsets,16],[__consumer_offsets,49],[TEST.RDM.TO.RDS.CORE.Organization.OrganizationIdentifier,0],[__consumer_offsets,44],[__consumer_offsets,28],[DEV.RDM.TO.RDS.CORE.Organization.Organization,0],[TEST.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnit,0],[__consumer_offsets,17],[TEST.RDM.TO.RDS.CORE.Location.LocationIdentifier,0],[__consumer_offsets,23],[__consumer_offsets,7],[TEST.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnitIdentifier,0],[DEV.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnitIdentifier,0],[__consumer_offsets,4],[__consumer_offsets,29],[__consumer_offsets,35],[__consumer_offsets,3],[__consumer_offsets,24],[__consumer_offsets,41],[TEST.RDM.TO.RDS.CORE.Organization.Organization,0],[__consumer_offsets,0],[__consumer_offsets,38],[__consumer_offsets,13],[TEST.RDM.TO.RDS.CORE.Location.ActivityCenterIdentifier,0],[DEV.RDM.TO.RDS.CORE.Organization.OrganizationIdentifier,0],[TEST.RDM.TO.RDS.CORE.Organization.OperationalActivityCenterIdentifier,0],[__consumer_offsets,8],[__consumer_offsets,5],[__consumer_offsets,39],[__consumer_offsets,36],[__consumer_offsets,40],[__consumer_offsets,45],[__consumer_offsets,15],[__consumer_offsets,33],[__consumer_offsets,37],[TEST.RDM.TO.RDS.CORE.Location.Location,0],[TEST.RDM.TO.RDS.CORE.Location.ActivityCenter,0],[__consumer_offsets,21],[__consumer_offsets,6],[TEST.RDM.TO.RDS.CORE.Organization.OperationalActivityCenter,0],[__consumer_offsets,11],[__consumer_offsets,20],[__consumer_offsets,47],[__consumer_offsets,2],[__consumer_offsets,27],[__consumer_offsets,34],[Golden.IMS.CDC.3NF.GLOBAL.RDM.TEST,0],[__consumer_offsets,9],[__consumer_offsets,22],[__consumer_offsets,42],[__consumer_offsets,14],[__consumer_offsets,25],[DEV.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnit,0],[__consumer_offsets,10],[__consumer_offsets,48],[__consumer_offsets,31],[__consumer_offsets,18],[__consumer_offsets,19],[__consumer_offsets,12],[TEST.RDM.TO.RDS.LOCAL.Organization.Outlet,0],[__consumer_offsets,46],[__consumer_offsets,43],[__consumer_offsets,1],[__consumer_offsets,26],[__consumer_offsets,30] (kafka.server.ReplicaFetcherManager)
[2016-12-08 11:22:01,118] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,25] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,128] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,25] in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,128] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,31] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,132] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,31] in 4 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,132] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,37] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,137] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,37] in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,137] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,43] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,142] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,43] in 4 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,142] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,49] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,145] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,49] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,146] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,44] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,150] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,44] in 4 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,150] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,1] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,155] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,1] in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,155] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,7] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,159] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,7] in 4 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,159] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,13] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,167] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,13] in 8 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,167] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,19] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,173] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,19] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,173] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,2] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,176] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,2] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,177] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,8] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,180] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [__consumer_offsets,32],[__consumer_offsets,16],[__consumer_offsets,49],[TEST.RDM.TO.RDS.CORE.Organization.OrganizationIdentifier,0],[__consumer_offsets,44],[__consumer_offsets,28],[DEV.RDM.TO.RDS.CORE.Organization.Organization,0],[TEST.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnit,0],[__consumer_offsets,17],[TEST.RDM.TO.RDS.CORE.Location.LocationIdentifier,0],[__consumer_offsets,23],[__consumer_offsets,7],[TEST.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnitIdentifier,0],[DEV.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnitIdentifier,0],[__consumer_offsets,4],[__consumer_offsets,29],[__consumer_offsets,35],[__consumer_offsets,3],[__consumer_offsets,24],[__consumer_offsets,41],[TEST.RDM.TO.RDS.CORE.Organization.Organization,0],[__consumer_offsets,0],[__consumer_offsets,38],[__consumer_offsets,13],[TEST.RDM.TO.RDS.CORE.Location.ActivityCenterIdentifier,0],[DEV.RDM.TO.RDS.CORE.Organization.OrganizationIdentifier,0],[TEST.RDM.TO.RDS.CORE.Organization.OperationalActivityCenterIdentifier,0],[__consumer_offsets,8],[__consumer_offsets,5],[__consumer_offsets,39],[__consumer_offsets,36],[__consumer_offsets,40],[__consumer_offsets,45],[__consumer_offsets,15],[__consumer_offsets,33],[__consumer_offsets,37],[TEST.RDM.TO.RDS.CORE.Location.Location,0],[TEST.RDM.TO.RDS.CORE.Location.ActivityCenter,0],[__consumer_offsets,21],[__consumer_offsets,6],[TEST.RDM.TO.RDS.CORE.Organization.OperationalActivityCenter,0],[__consumer_offsets,11],[__consumer_offsets,20],[__consumer_offsets,47],[__consumer_offsets,2],[__consumer_offsets,27],[__consumer_offsets,34],[Golden.IMS.CDC.3NF.GLOBAL.RDM.TEST,0],[__consumer_offsets,9],[__consumer_offsets,22],[__consumer_offsets,42],[__consumer_offsets,14],[__consumer_offsets,25],[DEV.RDM.TO.RDS.CORE.Organization.OrganizationOperationalUnit,0],[__consumer_offsets,10],[__consumer_offsets,48],[__consumer_offsets,31],[__consumer_offsets,18],[__consumer_offsets,19],[__consumer_offsets,12],[TEST.RDM.TO.RDS.LOCAL.Organization.Outlet,0],[__consumer_offsets,46],[__consumer_offsets,43],[__consumer_offsets,1],[__consumer_offsets,26],[__consumer_offsets,30] (kafka.server.ReplicaFetcherManager)
[2016-12-08 11:22:01,184] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,8] in 7 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,184] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,14] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,190] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,14] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,190] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,20] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,196] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,20] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,196] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,26] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,203] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,26] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,203] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,32] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,209] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,32] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,209] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,38] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,221] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,38] in 12 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,221] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,3] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,229] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,3] in 7 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,229] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,9] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,235] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,9] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,235] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,15] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,244] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,15] in 8 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,244] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,21] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,248] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,21] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,249] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,27] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,257] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,27] in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,257] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,33] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,265] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,33] in 8 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,266] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,39] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,273] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,39] in 7 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,273] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,45] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,280] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,45] in 7 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,280] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,22] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,287] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,22] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,287] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,28] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,295] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,28] in 8 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,295] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,34] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,305] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,34] in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,305] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,40] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,308] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,40] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,309] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,46] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,312] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,46] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,313] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,41] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,319] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,41] in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,319] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,47] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,324] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,47] in 5 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,324] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,4] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,332] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,4] in 7 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,332] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,10] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,339] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,10] in 7 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,340] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,16] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,346] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,16] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,346] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,5] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,356] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,5] in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,356] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,11] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,362] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,11] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,362] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,17] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,373] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,17] in 11 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,374] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,23] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,375] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,23] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,375] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,29] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,376] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,29] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,376] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,35] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,378] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,35] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,378] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,0] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,379] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,0] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,379] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,6] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,380] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,6] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,380] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,12] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,381] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,12] in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,382] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,18] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,384] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,18] in 2 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,385] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,24] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,396] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,24] in 11 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,397] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,30] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,400] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,30] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,401] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,36] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,404] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,36] in 3 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,405] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,42] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,417] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,42] in 2 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,417] INFO [Group Metadata Manager on Broker 0]: Loading offsets and group metadata from [__consumer_offsets,48] (kafka.coordinator.GroupMetadataManager)
[2016-12-08 11:22:01,419] INFO [Group Metadata Manager on Broker 0]: Finished loading offsets from [__consumer_offsets,48] in 2 milliseconds. (kafka.coordinator.GroupMetadataManager)
Update installed Kafka
--
If you already have Kafka installed and you want to update it using Homebrew.
Open terminal
$ brew update
Homebrew will update the core as well as the installed Formulae
Updated 1 tap (homebrew/core).
==> New Formulae
closure-stylesheets jsonlint mysql@5.6 stern
coffeescript juju@2.0 nanomsgxx swiftformat
dnsviz kops node@0.10 tarsnapper
eject libgc node@0.12 terraform-provisioner-ansible
elasticsearch@1.7 libgosu node@4 tomcat@6
elasticsearch@2.4 libosmium node@5 tomcat@7
github-keygen luaver node@6 treefrog
glfw lumo packetbeat ttygif
gmic metricbeat qwtpolar twarc
goenv mfterm recon-ng wdc
google-java-format mikutter sbjson wu
gst-rtsp-server minimesos snap-telemetry
ib mysql@5.5 sonarlint
==> Updated Formulae
acpica etcd jfrog-cli-go ncmpcpp sdl_image
advancecomp exploitdb jmeter neo4j sdl_mixer
afflib fabio jmxtrans neofetch sf-pwgen
agda fdroidserver jpeg-turbo neon sfk
aha feedgnuplot jruby netpbm sfml
algernon ffind jsdoc3 nexus shadowsocks-libev
ammonite-repl ffmpeg json-glib nghttp2 shairport-sync
android-ndk file-roller jsonschema2pojo nifi shared-mime-info
android-platform-tools filebeat kafka ✔ nim shellinabox
ansible fish kapacitor ninja sile
ansifilter flow kawa node sjk
antigen fltk kibana node-build skinny
apache-spark fluent-bit knot nomad slimerjs
apache-zeppelin fmt kotlin noti snappystream
app-engine-go-32 folly kubernetes-cli notmuch snapraid
app-engine-go-64 fontforge lablgtk nsd snzip
app-engine-java fonttools languagetool nss softhsm
arangodb freeipmi lbdb nuget sops
archi-steam-farm freeling lean-cli ocaml sourcekitten
argyll-cms freetds leaps ocamlbuild speedtest_cli
aria2 fstar ledit ocamlsdl sqldiff
armor fuseki lesspipe offlineimap sqlite
artifactory fwup lfe omega sqlite-analyzer
asio fzf libarchive oniguruma sqlmap
ats2-postiats galen libcdr ooniprobe srclib
augeas gdb libcec open-scene-graph sslscan
aws-elasticbeanstalk gecode libcouchbase openssl@1.1 ssreflect
aws-sdk-cpp gegl libcue openvdb stunnel
awscli generate-json-schema libdap openvpn stuntman
azure-cli gflags libebur128 orientdb subnetcalc
babl ghc libetonyek osm2pgrouting subversion
baobab gibo libexosip osm2pgsql supersonic
baresip git libgit2 osmosis suricata
bash git-annex libgpg-error osquery svgcleaner
bazel git-cola libgphoto2 osrm-backend svtplay-dl
beecrypt git-ftp libgsf packer swift
bibtexconv git-lfs libical paket swiftgen
bind git-octopus libical-glib pandoc swiftlint
bitrise git-series libicns pandoc-citeproc swig
boost-build gitbucket liblastfm pandoc-crossref sync_gateway
borg gitlab-ci-multi-runner libmicrohttpd parallel syncthing
bro gjstest libmill parquet-tools syncthing-inotify
buku glade libmspub passenger synfig
bullet glbinding libosip pazpar2 sysdig
bzrtools glib libphonenumber pcap_dnsproxy szl
cabal-install glm libpng pdf2htmlex tag
caf globus-toolkit libpqxx pdftoedn tasksh
cake glog libpst percona-server taylor
calabash glyr libraw percona-toolkit tbb
camlp4 gnome-builder librdkafka percona-xtrabackup tbox
camlp5 gnu-tar libre pex tccutil
cassandra go librem pg_top tcpreplay
certbot goaccess libressl pgcli td
certigo godep libsamplerate pgloader telegraf
ceylon gofabric8 libtiff pgpool-ii terraform
charm-tools google-benchmark libu2f-host pgroonga terragrunt
cheat gphoto2 libunistring pgrouting testssl
check_postgres gradle libuv pigz texmath
checkstyle grafana libvirt pldebugger tgui
chromedriver grails libvisio pmd the_silver_searcher
chruby-fish graphite2 libwpd pngquant tig
citus grib-api libxml++3 ponyc tile38
cli53 groonga libzdb poppler tinc
clib gspell links postgis tinyproxy
cliclick gst-editing-services llnode postgresql tippecanoe
clipper gst-libav loc postgrest tmux
clojurescript gst-plugins-bad log4cxx pre-commit tomcat
closure-compiler gst-plugins-base logstash prometheus tomcat-native
clutter-gst gst-plugins-good logtalk psql2csv tor
cmake gst-plugins-ugly luaradio psqlodbc trafficserver
cmark gst-python m-cli pugixml translate-shell
cmus gst-validate mackup pulledpork ttfautohint
collectd gstreamer macvim purescript ttyd
commandbox gtk+3 mailutils pushpin typesafe-activator
conan gtk-gnutella mandoc pwntools typescript
consul gtkdatabox mapnik pyenv u-boot-tools
consul-backinator gtkextra mariadb pypy ufraw
convox gtksourceview3 mawk pypy3 uncrustify
coq gucharmap mbedtls qbs urweb
corectl gupnp-tools mdds qjackctl vala
coreutils gwt mdp qjson vault
couchdb gwyddion mdv quantlib vc4asm
cpanminus haproxy media-info quazip vdirsyncer
cpprestsdk harfbuzz mediaconch quilt vim
cromwell headphones megatools qxmpp vmtouch
cryptopp hebcal memcached rakudo-star voldemort
crystal-lang heroku mercurial rancher-cli vte3
curl highlight meson rancher-compose vultr
czmq hledger mesos rancid wartremover
datetime-fortran homebank mg3a rbenv watson
davmail hunspell micro rclone webkitgtk
dbhash hyper micropython re2 wellington
dbus hyperscan minio readline winetricks
dcraw i2p minio-mc redis wireguard-tools
dependency-check icu4c minizinc rhash wireshark
diff-pdf imagejs mitmproxy riemann x264
digdag imagemagick mkdocs ripgrep xapian
distribution imapfilter mksh rocksdb xcproj
dmd infer mkvtoolnix roswell xctool
docker influxdb mlt rtags yarn
docker-compose innotop mmix ruby yaz
docker-machine-driver-xhyve internetarchive moc ruby-build yle-dl
docker-swarm ios-sim mongodb runit you-get
dovecot ipbt monkeysphere rust youtube-dl
dpkg iperf3 mono s-search z3
dub ipmitool mpd s6 zabbix
duplicity ironcli mpg123 saltstack zebra
dwdiff iso-codes mpop sassc zero-install
efl jadx mpv sbcl zeromq
eigen jasper msitools scala znapzend
elasticsearch javarepl msmtp scale2x zookeeper ✔
elixirscript jboss-forge mu scamper zorba
elm jemalloc mutt schismtracker zplug
emscripten jenkins mycli scons zsh-completions
ephemeralpg jenkins-job-builder mysql-sandbox scw zstd
eris jetty mytop sdl2 zurl
==> Renamed Formulae
juju -> juju@1.25 libmongoclient -> mongo-cxx-driver mongo-c -> mongo-c-driver
==> Deleted Formulae
bzr-explorer czmqpp libechonest libqglviewer polygen swish-e tlassemble wyrd
coin libbson libnice pdksh pyqt tevent ttylog yelp-xsl
==> Migrating HOMEBREW_REPOSITORY (please wait)...
==> Migrated HOMEBREW_REPOSITORY to /usr/local/Homebrew!
Homebrew no longer needs to have ownership of /usr/local. If you wish you can
return /usr/local to its default ownership with:
sudo chown root:wheel /usr/local
Updated formulae will be marked for easy identification.
This is an update to my previous blog: kafka on macos-Sierra
- Hardware
- Model Name: MacBook Pro
- Model Identifier: MacBookPro15,1
- Processor Name: Intel Core i9
- Processor Speed: 2.9 GHz
- Number of Processors: 1
- Total Number of Cores:6
- L2 Cache (per Core): 256 KB L3 Cache: 12 MB
- Memory: 32 GB
- Software
- OS: macOS 10.14
- Package Manager: Homebrew 1.7.6
Installation
I used Homebre to install Kafka.
$ brew search kafka
==> Formulae
kafka kafkacat librdkafka
==> Casks
homebrew/cask/kafka-tool
$
$ brew install kafka
==> Installing dependencies for kafka: zookeeper
==> Installing kafka dependency: zookeeper
==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.12.mojave.bottle.tar.gz
######################################################################## 100.0%
==> Pouring zookeeper-3.4.12.mojave.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink include/zookeeper
/usr/local/include is not writable.
You can try again using:
brew link zookeeper
==> Caveats
To have launchd start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
zkServer start
==> Summary
🍺 /usr/local/Cellar/zookeeper/3.4.12: 242 files, 32.9MB
==> Installing kafka
==> Downloading https://homebrew.bintray.com/bottles/kafka-2.0.0.mojave.bottle.tar.gz
######################################################################## 100.0%
==> Pouring kafka-2.0.0.mojave.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
🍺 /usr/local/Cellar/kafka/2.0.0: 160 files, 46.8MB
==> Caveats
==> zookeeper
To have launchd start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
zkServer start
==> kafka
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
$
Homebrew was not able to create the symlink. So, I tried creating it manually as suggested. That didn't work. Then I tried re-installing zookeeper. Still no luck in creating symlink.
Used brew doctor to diagonise the issue.
Warning: zookeeper 3.4.12 is already installed, it's just not linked
You can use `brew link zookeeper` to link this version.
$ brew link zookeeper
Linking /usr/local/Cellar/zookeeper/3.4.12...
Error: Could not symlink include/zookeeper
/usr/local/include is not writable.
$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!
Warning: The following directories do not exist:
/usr/local/include
/usr/local/sbin
/usr/local/Frameworks
You should create these directories and change their ownership to your account.
sudo mkdir -p /usr/local/include /usr/local/sbin /usr/local/Frameworks
sudo chown -R $(whoami) /usr/local/include /usr/local/sbin /usr/local/Frameworks
Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
zookeeper
Warning: Your Xcode is configured with an invalid path.
You should change it to the correct path:
sudo xcode-select -switch /Applications/Xcode.app
And found that the required drives are no more present. Hence followed the steps suggested by the doctor to remediate the issue at hand.
$ sudo mkdir -p /usr/local/include /usr/local/sbin /usr/local/Frameworks
Password:
$ sudo chown -R $(whoami) /usr/local/include /usr/local/sbin /usr/local/Frameworks
$ sudo xcode-select -switch /Applications/Xcode.app
Tried creating the symlinks again
$ brew link zookeeper
Linking /usr/local/Cellar/zookeeper/3.4.12... 13 symlinks created
Installation completed successfully!
Starting Kafka
Start Dependency:
In order to start zookeeper
$ zkserver start
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
Once the dependency is started,
Start Kafka: Used the brew services to start it.
Start Kafka: Used the brew services to start it.
$ brew services start kafka
==> Tapping homebrew/services
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-services'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 14 (delta 0), reused 8 (delta 0), pack-reused 0
Unpacking objects: 100% (14/14), done.
Tapped 1 command (44 files, 59.0KB).
==> Successfully started `kafka` (label: homebrew.mxcl.kafka)
Kinshuk Dutta
New York
