In Part 1, we create single machine that was running Kafka. Now let's do some horizontal scaling!
Step 1. Create new directory and initialize it with the box that we created in Part 1.
mkdir debianKafkaClusterNode2
cd debianKafkaClusterNode2
vagrant init debianKafkaClusterNode2 <path_to_the_box>/debian-kafka-cluster.box
Step 2. Edit generated Vagrant file (See Part 1 for details)
- Make sure that the memory is set to at least 2048
- Change the IP to be 192.168.33.11
Step 3. Start this box up and log in
vagrant up
vagrant ssh
Step 4. Configure Kafka.
Open $KAFKA_HOME/config/server.properties and set following values
broker.id=2
host.name=192.168.33.11
Now, repeat Steps 1-4 for number of boxes that you want to set up for your Kafka cluster. Don't forget to keep track of broker.id and IP (Step 2 and 4) - make sure they are unique!
After you successfully created n number of boxes, bring up your first Kafka cluster box that your created in Part 1. We shall refer to it as Node1
vagrant up
vagrant ssh
Start Zookeeper and Kafka on Node1
sudo $ZK_HOME/bin/zkServer.sh start
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
Start Kafka on rest of the nodes.
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
Congrats! You have Kafka cluster! Test it out by going to Node1 and adding few messages to the topic. Use Ctrl+C to exit.
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.33.10:9092, 192.168.33.11:9092 --topic test-topic
this
is
a
test
Test that you can retrieve the messages from some other node in the cluster
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic test-topic --from-beginning
As you may have noticed, we use only one Zookeeper! To add more while still following majority rule, edit $KAFKA_HOME/config/server.properties by setting zookeeper.connect to the list of appropriate machines. This has to be done on each server. Don't forget to change $ZK_HOME/conf/zoo.cfg as well as myid. See Part 1 for more details. For example for 3 machine set up:
zoo.cfg file
server.1=192.168.33.10:2888:3888
server.2=192.168.33.11:2888:3888
server.3=192.168.33.12:2888:3888
echo "2" > /var/zookeeper/data/myid
echo "3" > /var/zookeeper/data/myid
Just to make sure that we have everything at this point, let's shut everything down and start it back up:
On each VM.
exit
vagrant halt
vagrant up
vagrant ssh
(Start Zookeeper and Kafka)
sudo $ZK_HOME/bin/zkServer.sh start
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
Use commands to add topics and messages and read them. Have fun!
Showing posts with label zookeeper. Show all posts
Showing posts with label zookeeper. Show all posts
Tuesday, September 9, 2014
Apache Kafka. Set up. Writing to. Reading from. Monitoring. Part 1
Phew! That was a mouth full... but, basically what I am going to try to do here is:
First thing first... What is Apache Kafka? Straight from the source: "Apache Kafka is publish-subscribe messaging rethought as a distributed commit log." There are number of documents and guides out there that talk about the architecture and design, so I am not going to restate it here. What we want is some hands on experience with Kafka to test it out... in my case, it is much easier for me to learn things if I can play with them as I read about them.
In order to set up our cluster, let's use Vagrant to help us with the set up and configuration of VMs. This way, we can always reuse it one a different system and most importantly, we will be able to delete VM and start from scratch if we were to mess up without trying to break our heads if we removed something properly from the system or not.
If you never used Vagrant before, consider walking through Getting Started tutorial to truly appreciate it!
Anyway let's download VirtualBox and Vagrant and install them. Please follow installation guides...
Now, create a directory which would hold your VM instance and properly name it so that you could tell it apart from other VMs in the future.
In this case, we are going to use very basic debian since we won't need fancy GUI, etc.
config.vm.provider "virtualbox" do |vb|
# # Don't boot with headless mode
# vb.gui = true
#
# # Use VBoxManage to customize the VM. For example to change memory:
vb.customize ["modifyvm", :id, "--memory", "2048"]
end
Since we want for this VM to be able to communicate with other boxes and it might be a good idea to SSH to it from our host system, let's find config.vm.network in the Vagrantfile and uncomment it out. Here we are going to assign it IP: 192.168.33.10
# Create a private network, which allows host-only access to the machine
# using a specific IP.
config.vm.network "private_network", ip: "192.168.33.10"
Now we are ready to start the box!
Now log into the box
And install java and text editor
Install Zookeeper
wget http://apache.claz.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
mkdir /opt/zookeeper
tar -zxvf zookeeper-3.4.6.tar.gz --directory /opt/zookeeper
cp /opt/zookeeper/zookeeper-3.4.6/conf/zoo_sample.cfg /opt/zookeeper/zookeeper-3.4.6/conf/zoo.cfg
Configure Zookeeper by creating a directory for zookeeper data
mkdir -p /var/zookeeper/data
and specifying this directory in /opt/zookeeper/zookeeper-3.4.6/conf/zoo.cfg file
by editing dataDir property like so
dataDir=/var/zookeeper/data
In the same file add if not already there value for first zookeeper server. Specify IP of our newly created machine.
server.1=192.168.33.10:2888:3888
Specify myid so that the server would be able to identify itself in the zookeeper cluster
echo "1" > /var/zookeeper/data/myid
Configure Kafka
Edit server.properties files in /opt/kafka/kafka-0.8.0-beta1-src/config
Set following values by finding them in the above mentioned file and making sure they are uncommented as well
broker.id=1
host.name=192.168.33.10
zookeeper.connect=192.168.33.10:2181
It would be much easier to add Zookeeper and Kafka location to the PATH so we don't have to refer to it by the entire path, plus if you later decide to move them you won't have to change hardcoded path. To add them to your environmental variables create or edit if already exists .bash_profile file like so
vim ~/.bash_profile
add following entries to it
export ZK_HOME=/opt/zookeeper/zookeeper-3.4.6/
export KAFKA_HOME=/opt/kafka/kafka-0.8.0-beta1-src/
export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
Make sure to close and open new terminal window for changes to take effect!
Start Zookeeper
sudo $ZK_HOME/bin/zkServer.sh start
Start Kafka
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
Test Kafka by listing available topics. At first, it should not have any:
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.33.10:2181
Test Kafka by creating a new topic
$KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.33.10:2181 --replica 1 --partition 1 --topic test-topic
Re-run command to list available topics. You should see test-topic in the list
Test Kafka by producing massages tot hat topic from the console. Use ctrl+c when finished.
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.33.10:9092 --topic test-topic
This
is
Kafka
Test
Test Kafka consumer to verify that messages are there for that topic.
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic test-topic --from-beginning
You should see
This
is
Kafka
Test
If you followed above test steps and everything worked as expected, it is time to package this box so that you can re-use it for our future boxes within cluster. On your host machine execute
VBoxManage list vms
You should see something like this:
Use this name to package it up into a box.
vagrant package --base debian-cluster-node-1_default_1409266303013_22617 --output debian-kafka-cluster.box
Once Vagrant is done you should see debian-kafka-cluster.box in the same directory. Go ahead and shutdown the VM.
vagrant halt
In summary, we created a VM, installed all required software on it (java, zookeeper, kafka) and configured it to work within a cluster. Next we would have to duplicate our box into several machines so that we can have a Kafka cluster.
- Show you how to set up your own Apache Kafka cluster.
- Write to Apache Kafka and read from it using Kafka supplied shell scripts and then by using Java client.
- Finally, we quickly review KafkaOffsetMonitor - an app to monitor your Kafka consumers and their position (offset) in the queue.
First thing first... What is Apache Kafka? Straight from the source: "Apache Kafka is publish-subscribe messaging rethought as a distributed commit log." There are number of documents and guides out there that talk about the architecture and design, so I am not going to restate it here. What we want is some hands on experience with Kafka to test it out... in my case, it is much easier for me to learn things if I can play with them as I read about them.
In order to set up our cluster, let's use Vagrant to help us with the set up and configuration of VMs. This way, we can always reuse it one a different system and most importantly, we will be able to delete VM and start from scratch if we were to mess up without trying to break our heads if we removed something properly from the system or not.
If you never used Vagrant before, consider walking through Getting Started tutorial to truly appreciate it!
Anyway let's download VirtualBox and Vagrant and install them. Please follow installation guides...
Now, create a directory which would hold your VM instance and properly name it so that you could tell it apart from other VMs in the future.
In this case, we are going to use very basic debian since we won't need fancy GUI, etc.
mkdir debianKafkaClusterNode1
cd debianKafkaClusterNode1
vagrant init debianKafkaClusterNode1 http://puppet-vagrant-boxes.puppetlabs.com/debian-70rc1-x64-vbox4210.box config.vm.provider "virtualbox" do |vb|
# # Don't boot with headless mode
# vb.gui = true
#
# # Use VBoxManage to customize the VM. For example to change memory:
vb.customize ["modifyvm", :id, "--memory", "2048"]
end
Since we want for this VM to be able to communicate with other boxes and it might be a good idea to SSH to it from our host system, let's find config.vm.network in the Vagrantfile and uncomment it out. Here we are going to assign it IP: 192.168.33.10
# Create a private network, which allows host-only access to the machine
# using a specific IP.
config.vm.network "private_network", ip: "192.168.33.10"
Now we are ready to start the box!
vagrant up
The very first time it might take sometime since Vagrant will attempt to download
debian-70rc1-x64-vbox4210.boxfromhttp://puppet-vagrant-boxes.puppetlabs.com/.
Now log into the box
vagrant ssh
And install java and text editor
sudo apt-get update
sudo apt-get install openjdk-8-jdk
sudo apt-get install vim
Now we are going to install Apache Kafka by downloading its source code and building it.sudo su -
wget https://archive.apache.org/dist/kafka/kafka-0.8.0-beta1-src.tgz
mkdir /opt/kafka
tar -zxvf kafka-0.8.0-beta1-src.tgz
cd kafka-0.8.0-beta1-src
./sbt update
./sbt package
./sbt assembly-package-dependency
cd ../
mv kafka-0.8.0-beta1-src /opt/kafka
Install Zookeeper
wget http://apache.claz.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
mkdir /opt/zookeeper
tar -zxvf zookeeper-3.4.6.tar.gz --directory /opt/zookeeper
cp /opt/zookeeper/zookeeper-3.4.6/conf/zoo_sample.cfg /opt/zookeeper/zookeeper-3.4.6/conf/zoo.cfg
Configure Zookeeper by creating a directory for zookeeper data
mkdir -p /var/zookeeper/data
and specifying this directory in /opt/zookeeper/zookeeper-3.4.6/conf/zoo.cfg file
by editing dataDir property like so
dataDir=/var/zookeeper/data
In the same file add if not already there value for first zookeeper server. Specify IP of our newly created machine.
server.1=192.168.33.10:2888:3888
Specify myid so that the server would be able to identify itself in the zookeeper cluster
echo "1" > /var/zookeeper/data/myid
Configure Kafka
Edit server.properties files in /opt/kafka/kafka-0.8.0-beta1-src/config
Set following values by finding them in the above mentioned file and making sure they are uncommented as well
broker.id=1
host.name=192.168.33.10
zookeeper.connect=192.168.33.10:2181
It would be much easier to add Zookeeper and Kafka location to the PATH so we don't have to refer to it by the entire path, plus if you later decide to move them you won't have to change hardcoded path. To add them to your environmental variables create or edit if already exists .bash_profile file like so
vim ~/.bash_profile
add following entries to it
export ZK_HOME=/opt/zookeeper/zookeeper-3.4.6/
export KAFKA_HOME=/opt/kafka/kafka-0.8.0-beta1-src/
export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
Make sure to close and open new terminal window for changes to take effect!
Start Zookeeper
sudo $ZK_HOME/bin/zkServer.sh start
Start Kafka
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
Test Kafka by listing available topics. At first, it should not have any:
$KAFKA_HOME/bin/kafka-list-topic.sh --zookeeper 192.168.33.10:2181
Test Kafka by creating a new topic
$KAFKA_HOME/bin/kafka-create-topic.sh --zookeeper 192.168.33.10:2181 --replica 1 --partition 1 --topic test-topic
Re-run command to list available topics. You should see test-topic in the list
Test Kafka by producing massages tot hat topic from the console. Use ctrl+c when finished.
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.33.10:9092 --topic test-topic
This
is
Kafka
Test
Test Kafka consumer to verify that messages are there for that topic.
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic test-topic --from-beginning
You should see
This
is
Kafka
Test
If you followed above test steps and everything worked as expected, it is time to package this box so that you can re-use it for our future boxes within cluster. On your host machine execute
VBoxManage list vms
You should see something like this:
"debian-cluster-node-1_default_1409266303013_22617" {3d996de2-94e1-4d72-be8f-29f36150ac84}
Use this name to package it up into a box.
vagrant package --base debian-cluster-node-1_default_1409266303013_22617 --output debian-kafka-cluster.box
Once Vagrant is done you should see debian-kafka-cluster.box in the same directory. Go ahead and shutdown the VM.
vagrant halt
In summary, we created a VM, installed all required software on it (java, zookeeper, kafka) and configured it to work within a cluster. Next we would have to duplicate our box into several machines so that we can have a Kafka cluster.
Saturday, August 16, 2014
How to create SolrCloud Instance.
In this post, I decided to talk about how to quickly get SolrCloud instance up and running on your local box.
First of all, make sure that you have Java 6+ installed on your system.
First of all, make sure that you have Java 6+ installed on your system.
$ java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
Download most recent version of Solr from http://lucene.apache.org/solr/. If you are on Unix, Linux or Mac OS, grab tgz file. Download it somewhere where you would have permissions to untar it and to run it. I downloaded and decompressed it under my home directory in /opt/solr by running:
tar solr-<version>.tgz
In this example, I am going to be using default jetty server, which was proven to scale even on large production systems and since I don't want to further complicate this example. If you were to google "sold jetty vs tomcat", you would see tons of opinions out there on which one to use. In my opinion, use whatever makes more sense in your case.
To run very basic Solr instance, all you have to do is to navigate to example folder and to run start.jar like so
$ cd opt/solr/example/
$ java -jar start.jar
However, we are interested in SolrCloud Instance in this case... so let's create two shards with replication of 2. For detailed discussion of shards, replication and SolrCloud in general please visit SolrCloud wiki.
Copy example directory and name it node1. After that rename collection name to something more useful, like wikipedia. Remove any data that might be there. And do some autodiscovery magic.
$ cp -r example/ node1/
$ cd node1
$ cp -r solr/collection1/ solr/wikipedia
$ rm -rf solr/wikipedia/data/
$ find . -name "core.properties" -type f -exec rm {} \;
$ echo "name=wikipedia" > solr/wikipedia/core.properties
Now you are ready to start your first node in your SolrCloud
$ java -Dcollection.configName=wikipedia -DzkRun -DnumShards=2 -Dbootstrap_confdir=./solr/wikipedia/conf/ -jar start.jar
Now let's create 3 more nodes and start them to complete our SolrCloud Instance
$ cp -r node1/ node2/
$ cd node2/
$ rm -rf solr/wikipedia/conf/
$ java -DzkHost=localhost:9983 -Djetty.port=8984 -jar start.jar
$ cp -r node1/ node3/
$ cd node3/
$ rm -rf solr/wikipedia/conf/
$ java -DzkHost=localhost:9983 -Djetty.port=8985 -jar start.jar
$ cp -r node1/ node4/
$ cd node4/
$ rm -rf solr/wikipedia/conf/
$ java -DzkHost=localhost:9983 -Djetty.port=8986 -jar start.jar
Since all of our configuration files are now managed by Zookeeper, we would need to download them, then modify them and upload them back... yes, I know it is a pain, but this way you don't have to do it manually on each server, just do it once, upload it to Zookeeper and it would take care of the rest for you!
Navigate to /opt/solr/node1/scripts/cloud-scripts directory and run following command
$ ./zkcli.sh -zkhost localhost:9983 -cmd downconfig -confdir /<directory_of_your_choice>/solr_conf -confname wikipedia
Navigate to the directory and you should see all of the configuration files.
$ ls
_schema_analysis_stopwords_english.json elevate.xml solrconfig.xml
_schema_analysis_synonyms_english.json lang spellings.txt
admin-extra.html mapping-FoldToASCII.txt stopwords.txt
admin-extra.menu-bottom.html mapping-ISOLatin1Accent.txt synonyms.txt
admin-extra.menu-top.html protwords.txt update-script.js
clustering schema.xml velocity
currency.xml scripts.conf xslt
Make changes (usually to schema.xmla nd to solrconfig.xml) and upload it back by running similar command.
$ ./zkcli.sh -zkhost localhost:9983 -cmd upconfig -confdir /<directory_of_your_choice>/solr_conf -confname wikipedia
That's it!
Thursday, July 31, 2014
What is Overseer in SolrCloud
This is by far the best and to the point explanation that I was able to find. Credit to Mark Miller.
The Overseer isn't mentioned much because it's an implementation
detail that the user doesn't have to really consider.
The Overseer first came about to handle writing the clusterstate.json
file, as a suggestion by Ted Dunning.
Originally, each node would try and update the custerstate.json file
themselves - and use optimistic locking and retries.
We decided that a cleaner method was to have an overseer and let new
nodes register themselves and their latest state as part of a list -
the Overseer then watches this list, and when things change, publishes
a new clusterstate.json - no optimistic locking and retries needed.
All the other nodes watch clusterstate.json and are notified to
re-read it when it changes.
Since, the Overseer has picked up a few other duties when it makes
sense. For example, it handles the shard assignments if a user does
not specify them. It also does the work for the collections api -
eventually this will be beneficial in that it will use a distributed
work queue and be able to resume operations that fail before
completing.
I think over time, there are lots of useful applications for the Overseer.
He is elected in the same manner as a leader for a shard - if the
Overseer goes down, someone simply takes his place.
I don't think the Overseer is going away any time soon.
- Mark
The Overseer isn't mentioned much because it's an implementation
detail that the user doesn't have to really consider.
The Overseer first came about to handle writing the clusterstate.json
file, as a suggestion by Ted Dunning.
Originally, each node would try and update the custerstate.json file
themselves - and use optimistic locking and retries.
We decided that a cleaner method was to have an overseer and let new
nodes register themselves and their latest state as part of a list -
the Overseer then watches this list, and when things change, publishes
a new clusterstate.json - no optimistic locking and retries needed.
All the other nodes watch clusterstate.json and are notified to
re-read it when it changes.
Since, the Overseer has picked up a few other duties when it makes
sense. For example, it handles the shard assignments if a user does
not specify them. It also does the work for the collections api -
eventually this will be beneficial in that it will use a distributed
work queue and be able to resume operations that fail before
completing.
I think over time, there are lots of useful applications for the Overseer.
He is elected in the same manner as a leader for a shard - if the
Overseer goes down, someone simply takes his place.
I don't think the Overseer is going away any time soon.
- Mark
Tuesday, March 18, 2014
Installing Zookeeper
Think of ZooKeeper as coordination service for cluster environment managing distributed processes.
ZooKeeper ensures:
- Consistency
- Atomicity
- Reliability
- Timeliness
In order to install ZooKeeper, you need to download stable release from its official site, extract it and make sure that our Hadoop user can access it.
[root@localhost local]# cd /usr/local/
[root@localhost local]# wget http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
--2014-03-03 20:25:54-- http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
Resolving apache.osuosl.org... 140.211.166.134
Connecting to apache.osuosl.org|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17699306 (17M) [application/x-gzip]
Saving to: “zookeeper-3.4.6.tar.gz”
100%[======================>] 17,699,306 280K/s in 45s
2014-03-03 20:26:41 (387 KB/s) - “zookeeper-3.4.6.tar.gz” saved [17699306/17699306]
[root@localhost local]# tar xzf zookeeper-3.4.6.tar.gz
[root@localhost local]# mv zookeeper-3.4.6 zookeeper
[root@localhost local]# chown -R hduser:hadoop zookeeper
[root@localhost local]# ll
….
drwxr-xr-x. 10 hduser hadoop 4096 Feb 20 05:58 zookeeper
…..
[root@localhost local]# cat /usr/local/zookeeper/conf/zoo_sample.cfg >> /usr/local/zookeeper/conf/zoo.cfg
It should look like
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
Notice that ZooKeeper needs temporary directory to store its files
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
Let’s change dataDir to point to /app/zookeeper and make sure that we have this directory with the right access rights.
[root@localhost app]# mkdir -p /app/zookeeper
[root@localhost app]# chown hduser:hadoop /app/zookeeper/
Now let’s start ZooKeeper
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
And verify that it is indeed running
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
2014-03-03 20:53:21,869 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-03-03 20:53:21,881 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost.localdomain
2014-03-03 20:53:21,882 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper/bin/../zookeeper-3.4.6.jar:/usr/local/zookeeper/bin/../src/java/lib/*.jar:/usr/local/zookeeper/bin/../conf:
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.5.1.el6.x86_64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hduser
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/hduser
2014-03-03 20:53:21,887 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/tmp
2014-03-03 20:53:21,891 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1d672476
Welcome to ZooKeeper!
2014-03-03 20:53:22,255 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2014-03-03 20:53:22,293 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2014-03-03 20:53:22,448 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x1448aca9c6e0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Now let’s set up ZOOKEEPER_HOME
$ sudo vim /etc/profile.d/zookeeper.sh
export ZOOKEEPER_HOME= /usr/local/zookeeper/
Done!
ZooKeeper ensures:
- Consistency
- Atomicity
- Reliability
- Timeliness
In order to install ZooKeeper, you need to download stable release from its official site, extract it and make sure that our Hadoop user can access it.
[root@localhost local]# cd /usr/local/
[root@localhost local]# wget http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
--2014-03-03 20:25:54-- http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
Resolving apache.osuosl.org... 140.211.166.134
Connecting to apache.osuosl.org|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17699306 (17M) [application/x-gzip]
Saving to: “zookeeper-3.4.6.tar.gz”
100%[======================>] 17,699,306 280K/s in 45s
2014-03-03 20:26:41 (387 KB/s) - “zookeeper-3.4.6.tar.gz” saved [17699306/17699306]
[root@localhost local]# tar xzf zookeeper-3.4.6.tar.gz
[root@localhost local]# mv zookeeper-3.4.6 zookeeper
[root@localhost local]# chown -R hduser:hadoop zookeeper
[root@localhost local]# ll
….
drwxr-xr-x. 10 hduser hadoop 4096 Feb 20 05:58 zookeeper
…..
Configuring ZooKeeper
Simply copy over sample config file to zoo.cfg[root@localhost local]# cat /usr/local/zookeeper/conf/zoo_sample.cfg >> /usr/local/zookeeper/conf/zoo.cfg
It should look like
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
Notice that ZooKeeper needs temporary directory to store its files
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
Let’s change dataDir to point to /app/zookeeper and make sure that we have this directory with the right access rights.
[root@localhost app]# mkdir -p /app/zookeeper
[root@localhost app]# chown hduser:hadoop /app/zookeeper/
Now let’s start ZooKeeper
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
And verify that it is indeed running
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
2014-03-03 20:53:21,869 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-03-03 20:53:21,881 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost.localdomain
2014-03-03 20:53:21,882 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper/bin/../zookeeper-3.4.6.jar:/usr/local/zookeeper/bin/../src/java/lib/*.jar:/usr/local/zookeeper/bin/../conf:
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.5.1.el6.x86_64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hduser
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/hduser
2014-03-03 20:53:21,887 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/tmp
2014-03-03 20:53:21,891 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1d672476
Welcome to ZooKeeper!
2014-03-03 20:53:22,255 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2014-03-03 20:53:22,293 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2014-03-03 20:53:22,448 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x1448aca9c6e0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Now let’s set up ZOOKEEPER_HOME
$ sudo vim /etc/profile.d/zookeeper.sh
export ZOOKEEPER_HOME= /usr/local/zookeeper/
Done!
Subscribe to:
Posts (Atom)