Getting Ambari up and running (with Vagrant)

I assume that you already have VirtualBox and Vagrant installed...

Set up

On guest machine, create a folder that would contain files for the VM
mkdir hadoop_ambari

Change to it and issue command to download the VM box and to add it to your library of VMs with specific name.
cd hadoop_ambari
vagrant box add hadoop_ambari https://github.com/2creatives/vagrant-centos/releases/download/v6.5.1/centos65-x86_64-20131205.box

Once download is complete, we can initialize Vagrant, which in turn would create Vagrantfile that acts as a configuration file. Various options are available to be specified: memory, ip, ports, etc.
vagrant init hadoop_ambari

Open Vagrantfile with vi editor and make following changes
config.vm.network :forwarded_port, guest: 8080, host: 8080
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provider "virtualbox" do |vb|  #
     # Use VBoxManage to customize the VM. For example to change memory:
     vb.customize ["modifyvm", :id, "--memory", "8192"]
end

Here we assigned static ip, opened port 8080 and made sure that we have 8GB of memory allocated to the machine.

Save the file and start vagrant
vagrant up

Once it starts, log in and change to root
vagrant ssh

inside of guest OS
sudo su
cd ~

Find out hostname of the machine
hostname

Edit /etc/hosts file to the following
vi /etc/hosts

192.168.33.11 <hostname>

(this ip was specified as static inside of your Vagrant file)

Install NTP service
yum install ntp

Install wget Utility
yum install wget

Turn on NTP service
chkconfig ntpd on
service ntpd start

Set up passwordless SSH
ssh-keygen
cd .ssh
cp id_rsa /vagrant
cat id_rsa.pub >> authorized_keys

Setup Ambari
wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.3.38/ambari.repo
cp ambari.repo /etc/yum.repos.d

Double check that repo was created
yum repolist

Install Ambari server
yum install ambari-server

Configure it (go with defaults)
ambari-server setup

Start Ambari Server
ambari-server start

Wait a little and you should be able to access your server at http://192.168.33.11:8080. Username and password: ambari/ambari. Follow the wizard which is self explanatory. In Install Options specify hostname of guest machine and then provide ssh private key by navigating to the hadoop_ambari folder that contains your Vagrantfile and id_rsa (remember when you copied your id_rsa file in guest OS to /vagrant folder?). In Customize Services, pick passwords for the services.

Wait for the installation to finish and enjoy your new set up! When browsing URLs inside of ambari, it by default would try to link to hostname and that won't work, so use the static ip instead. For example, MR2 JobHistory UI would be on http://192.168.33.11:19888.

Like always, comments? questions? just post!

Troubleshooting:

1.

If you can not get to the URLs, try to disable iptables
service iptables stop

Verify that curl is able to access the apache test page from inside the vm
vagrant ssh
curl -v localhost
If that doesn't work, then it's definitely not the port forwarding.

Lastly verify that the Host can access the page through curl
curl -v 'http://localhost:8080'

2.

Check if server is listening on 8080 and where it is binding
netstat -ntlp | grep 8080

Note that 127.0.0.1 is only accessible to the local machine, which for a guest machine means nothing.  Outside of the VM, it can't reach it! 0.0.0.0 is accessible from anywhere on the local network, which to a VM includes the host machine.

127.0.0.1 is normally the IP address assigned to the "loopback" or local-only interface. This is a "fake" network adapter that can only communicate within the same host. It's often used when you want a network-capable application to only serve clients on the same host. A process that is listening on 127.0.0.1 for connections will only receive local connections on that socket.

"localhost" is normally the hostname for the 127.0.0.1 IP address. It's usually set in /etc/hosts (or the Windows equivalent named "hosts" somewhere under %WINDIR%). You can use it just like any other hostname - try "ping localhost" to see how it resolves to 127.0.0.1.

0.0.0.0 has a couple of different meanings, but in this context, when a server is told to listen on 0.0.0.0 that means "listen on every available network interface". The loopback adapter with IP address 127.0.0.1 from the perspective of the server process looks just like any other network adapter on the machine, so a server told to listen on 0.0.0.0 will accept connections on that interface too.

Resources:

List of ports

  config.vm.network :forwarded_port, guest: 80, host: 42080, auto_correct: true #Apache http
  config.vm.network :forwarded_port, guest: 111, host: 42111, auto_correct: true #NFS portmap
  config.vm.network :forwarded_port, guest: 2223, host: 2223, auto_correct: true #Gateway node
  config.vm.network :forwarded_port, guest: 8000, host: 8000, auto_correct: true #Hue
  config.vm.network :forwarded_port, guest: 8020, host: 8020, auto_correct: true #Hdfs
  config.vm.network :forwarded_port, guest: 8042, host: 8042, auto_correct: true #NodeManager
  config.vm.network :forwarded_port, guest: 8050, host: 8050, auto_correct: true #Resource manager
  config.vm.network :forwarded_port, guest: 8080, host: 8080, auto_correct: true #Ambari
  config.vm.network :forwarded_port, guest: 8088, host: 8088, auto_correct: true #Yarn RM
  config.vm.network :forwarded_port, guest: 8443, host: 8443, auto_correct: true #Knox gateway
  config.vm.network :forwarded_port, guest: 8744, host: 8744, auto_correct: true #Storm UI
  config.vm.network :forwarded_port, guest: 8888, host: 8888, auto_correct: true #Tutorials
  config.vm.network :forwarded_port, guest: 10000, host: 10000, auto_correct: true #HiveServer2 thrift
  config.vm.network :forwarded_port, guest: 10001, host: 10001, auto_correct: true #HiveServer2 thrift http
  config.vm.network :forwarded_port, guest: 11000, host: 11000, auto_correct: true #Oozie
  config.vm.network :forwarded_port, guest: 15000, host: 15000, auto_correct: true #Falcon
  config.vm.network :forwarded_port, guest: 19888, host: 19888, auto_correct: true #Job history
  config.vm.network :forwarded_port, guest: 50070, host: 50070, auto_correct: true #WebHdfs
  config.vm.network :forwarded_port, guest: 50075, host: 50075, auto_correct: true #Datanode
  config.vm.network :forwarded_port, guest: 50111, host: 50111, auto_correct: true #WebHcat
  config.vm.network :forwarded_port, guest: 60080, host: 60080, auto_correct: true #WebHBase

References:

https://github.com/petro-rudenko/bigdata-toolbox/blob/master/Vagrantfile
http://serverfault.com/questions/513654/troubleshooting-why-1-vagrant-works-but-another-does-not
http://stackoverflow.com/questions/5984217/vagrants-port-forwarding-not-working
http://stackoverflow.com/questions/23840098/empty-reply-from-server-cant-connect-to-vagrant-vm-w-port-forwarding
http://stackoverflow.com/questions/20778771/what-is-the-difference-between-0-0-0-0-127-0-0-1-and-localhost

How to enable backspace in vim on Mac OS

Create a file called ~/.vimrc and put the following lines in it.

set nocompatible
set backspace=indent,eol,start

ENJOY!

How to customize JA-SIG CAS authentication process

Some people see Central Authentication Service (CAS) as a black box that magically tells you if you are authenticated against certain accreditation or not... Configuration is dead simple. All you have to do is to specify chain of authentication mechanisms in /src/main/webapp/WEB-INF/deployerConfigContext.xml file. CAS would try your login credentials against each one and eventually would either fail or pass.

Today, I would like to lift some of this mystery out and tell you how to customize parts of JA-SIG CAS to fit your requirements.

Here is the scenario, you are trying to authenticate via CAC card (X509 certificate) and need additional validation after user keys in his PIN. Let's say that we need to verify that the user is a part of certain organization that is listed in his certificate.

Step 1:
Create new Maven Web Application project

Step 2:
Add CAS dependencies to pom.xml file

        <!-- Main guts of CAS -->
         <dependency>
            <groupId>org.jasig.cas</groupId>
            <artifactId>cas-server-webapp</artifactId>
            <version>${cas.version}</version>
            <type>war</type>
            <scope>runtime</scope>
        </dependency>

        <!-- Needed to interact with X509 certificates -->

        <dependency>

            <groupId>org.jasig.cas</groupId>     

            <artifactId>cas-server-support-x509</artifactId>     

            <version>${cas.version}</version>

        </dependency>

Step 3:

Clean and build the project. It would download all the dependencies for your project.

Step 4: 

Add following line under 

<bean id="authenticationManager" class="org.jasig.cas.authentication.AuthenticationManagerImpl"> 

    <property name="credentialsToPrincipalResolvers">

         <list>

in your /src/main/webapp/WEB-INF/deployerConfigContext.xml configuration file.

<bean  class=  "org.jasig.cas.adaptors.x509.authentication.principal.X509CertificateCredentialsToIdentifierPrincipalResolver">

</bean>

Per documentation, "This is the List of CredentialToPrincipalResolvers that identify what Principal is trying to authenticate. The AuthenticationManagerImpl considers them in order, finding a CredentialToPrincipalResolver which supports the presented credentials."

This would basically tell our CAS to invoke X509CertificateCredentialsToIdentifierPrincipalResolver

Step 5:
Add following line under

<bean id="authenticationManager" class="org.jasig.cas.authentication.AuthenticationManagerImpl"> 

    <property name="authenticationHandlers">

         <list>

in the same configuration file.

    <bean class="org.jasig.cas.adaptors.x509.authentication.handler.support.X509CredentialsAuthenticationHandler">
        <property name="trustedIssuerDnPattern" value=".*, OU=<replace it>, O=<replace it>, C=<replace it>" />
        <property name="subjectDnPattern"  value=".*, OU=<replace it>, O=<replace it>, C=<replace it>" />
    </bean>

Per documentation: "Whereas CredentialsToPrincipalResolvers identify who it is some Credentials might authenticate, AuthenticationHandlers actually authenticate credentials.  Here we declare the AuthenticationHandlers that authenticate the Principals that the CredentialsToPrincipalResolvers identified.  CAS will try these handlers in turn until it finds one that both supports the Credentials presented and succeeds in authenticating."

Step 6:
Let's review... We have a new project that has all CAS dependencies in it and is configured to authenticate against CAC card by using org.jasig.cas.adaptors.x509.authentication.handler.support.X509CredentialsAuthenticationHandler

Step 7.
All JA-SIG CAS is public! So don't be afraid to review it! Get the latest release via git or download specific release that you perhaps specified in the pom file of our project.
In our case we are interested in

• org.jasig.cas.adaptors.x509.authentication.principal.X509CertificateCredentialsToIdentifierPrincipalResolver
• org.jasig.cas.adaptors.x509.authentication.handler.support.X509CredentialsAuthenticationHandler

Step 8.
X509CertificateCredentialsToIdentifierPrincipalResolver - is pretty basic and does not contain an actual logic of authentication. Remember? "Whereas CredentialsToPrincipalResolvers identify who it is some Credentials might authenticate, AuthenticationHandlers actually authenticate credentials."
So let's look at X509CredentialsAuthenticationHandler

Step 9.
X509CredentialsAuthenticationHandler contains
protected final boolean doAuthentication(final Credentials credentials)
method that performs actual authentication and decides if user will be permitted to access the system or not. Here is an extract of the code that is self explanatory:
     
       if (valid && hasTrustedIssuer && clientCert != null) {
      x509Credentials.setCertificate(clientCert);
      this.log.info("Successfully authenticated " + credentials);
      return true;
       }
       this.log.info("Failed to authenticate " + credentials);
       return false;

Step 10.
So what if you were to asked to change the default behavior of X509CredentialsAuthenticationHandler in terms what it does to authenticate or if you were asked to extend it? Perhaps upon successful X509 authentication, you were required to check if specific user was a part of some group inside of LDAP? What to do?

Step 11.
In your Source Packages, create a new package that would be EXACTLY the same as X509CredentialsAuthenticationHandler so let's create org.jasig.cas.adaptors.x509.authentication.handler.support package.
Now, you need to add X509CredentialsAuthenticationHandler.java to this package, or you can rename it to something else to distinguish it from the "original" implementation. If you do decide to rename it, please make sure to update its name in /src/main/webapp/WEB-INF/deployerConfigContext.xml (Step 5).

Step 12.
Copy the source code of the X509CredentialsAuthenticationHandler and paste it into your new java class. Alter doAuthentication method to contact external LDAP server with extracted IssuerDN (or whatever) to check for specific group membership or any other requirement you might have.

Step 13.
Alter the last step where doAuthentication method returns authentication pass or fail flag (See step 9) to whatever you see fit your requirements.

Step 14.
Clean and Build again for changes to take effect.

Step 15.
Deploy generated cas.war file onto Tomcat or any other server, and try it out!
Please note, that to be able to generate cas.war after the build, you need to add

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-war-plugin</artifactId>
                <configuration>
                    <warName>cas</warName>
                </configuration>
            </plugin>
        </plugins>
    </build>

to your pom.xml file.




Hopefully this was useful to you and you are not feel more comfortable working and modifying internals of CAS. Like always feel free to reach out to me with questions.

Apache Kafka. Set up. Writing to. Reading from. Monitoring. Part 4

Now that we have Kafka cluster up and running (Part 1 Part 2), and we are able to monitor it (Part 3), we need to learn how to write to and read from it by using Java.

I have created and uploaded my projects onto github so feel free to download the code and follow along. It is configured to work with IP addresses and topics that we created and configured in Part 1 and 2.

Producer code
First you need to specify connection information. This is as simple as specifying broker list IPs and ports.
// List of brokers that the producer will try to connect to
props.put("metadata.broker.list", "192.168.33.21:9092,192.168.33.22:9092");

Inside of properties you can set number of different choices and flags. Later you would use it to create ProducerConfig object that would in turn be used to create Producer object that would be able to send out messages to Kafka cluster.

Then select a topic that you want to write to
String topic = "my_test_topic" ;

Construct a message that would contain your topic, message and id and use Producer object to send it.

String msg = "Test message # " + 1 ;
KeyedMessage<String, String> data = new KeyedMessage<String, String>(topic, String.valueOf(1), msg);
producer.send(data);

Consumer code
This code is a little bit more complex and I had to use lots of tricks to make sure that I will be able to read from a topic of interest. Basically, if you were not careful with offsets, you would be able to read anything... I tried to comment the code as much as possible so that it would be self explanatory, but the best way to understand it all is by running it with debug and see the process flow.

Hope you enjoyed this start to end Kafka cluster guide and don't hesitate to reach out to me with comments or questions!

Apache Kafka. Set up. Writing to. Reading from. Monitoring. Part 3

Now that we were able to communicate with our Kafka cluster by writing to and reading from it, we might be curious what we have there. What brokers we have, what are the offsets on different topics, etc.

The best way to grasp the big picture is tool that can give you nice graphical interface that is lightweight. I was able to find and get KafkaOffsetMonitor working on my cluster.

The instructions on the website are pretty easy to follow along so I won't repeat them here. Main point of this point is to make reader aware of KafkaOffsetMonitor tool to give start to end hands on experience with Kafka.

Enjoy!

SVN to GIT Migration

You need to move your code from SVN to GIT. What do you need to do?

First of all, get yourself familiar with the process and what steps you would need to do: https://www.atlassian.com/pt/git/migration#!migration-overview. In this tutorial, I will use a svn2git tool to help me with migration, but svn2git is a wrapper of git svn clone, so all of the basics would still apply. Now you are wondering why svn2git? Well, when I was using git svn clone described in the above mentioned article, I ran into the issue where SVN had spaces in tags, and git replaced them with %20 and that broke things. Go figure. Here is the issue. So after some digging and trying to resolve it, I came across svn2git project that had a work around for this issue

Download handy svn migration scripts from https://bitbucket.org/atlassian/svn-migration-scripts/downloads. We would use it to generate authors.txt file. You can find more details here.

After you have authors.txt file in hand, let's go ahead and make sure that we can run svn2git. I am on Mac, for Debian-based system refer to the svn2git installation guide.
Run following commands to make sure that you have git-core, git-svn, ruby and rubygems installed on your system. You should have them if you have Xcode installed. If not, install it!
git --help
git svn --help
ruby --help
gem --help

With help of rubygems install svn2get. This would also add it to your PATH
sudo gem install svn2git

Create new directory where you want your converted files to be stored. This directory will become your new local git repository.
mkdir gitcode
cd gitcode
svn2git http://svn.repo.com/path/to/repo --authors /path/to/authors/authors.txt --verbose

Refer to the svn2git installation guide for more options. In my case, I received
'master' did not match any file(s) known to git.
error and had to tweak my command slightly to make it work. Like so,
svn2git http://svn.repo.com/path/to/repo --authors /path/to/authors/authors.txt --verbose --trunk / --nobranches --notags
Basically, my SVN was not properly set up and I had to manually specify trunk location, and there were not branches or tags.

Depending on how much source code you have it can take a while... when it is all done: review the code and push it to a remote repository where everyone will be able to access it.
git remote add origin ssh://server/path/to/repo
git push origin master

That's it!

Apache Kafka. Set up. Writing to. Reading from. Monitoring. Part 2

In Part 1, we create single machine that was running Kafka. Now let's do some horizontal scaling!

Step 1. Create new directory and initialize it with the box that we created in Part 1.
mkdir debianKafkaClusterNode2
cd debianKafkaClusterNode2
vagrant init debianKafkaClusterNode2 <path_to_the_box>/debian-kafka-cluster.box

Step 2. Edit generated Vagrant file (See Part 1 for details)
- Make sure that the memory is set to at least 2048
- Change the IP to be 192.168.33.11

Step 3. Start this box up and log in
vagrant up
vagrant ssh

Step 4. Configure Kafka.
Open $KAFKA_HOME/config/server.properties and set following values
broker.id=2
host.name=192.168.33.11

Now, repeat Steps 1-4 for number of boxes that you want to set up for your Kafka cluster. Don't forget to keep track of broker.id and IP (Step 2 and 4) - make sure they are unique!

After you successfully created n number of boxes, bring up your first Kafka cluster box that your created in Part 1. We shall refer to it as Node1
vagrant up
vagrant ssh

Start Zookeeper and Kafka on Node1
sudo $ZK_HOME/bin/zkServer.sh start
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

Start Kafka on rest of the nodes.
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

Congrats! You have Kafka cluster! Test it out by going to Node1 and adding few messages to the topic. Use Ctrl+C to exit.
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.33.10:9092, 192.168.33.11:9092 --topic test-topic
this 
is
a 
test

Test that you can retrieve the messages from some other node in the cluster
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic test-topic --from-beginning

As you may have noticed, we use only one Zookeeper! To add more while still following majority rule, edit $KAFKA_HOME/config/server.properties by setting zookeeper.connect to the list of appropriate machines. This has to be done on each server. Don't forget to change $ZK_HOME/conf/zoo.cfg as well as myid. See Part 1 for more details. For example for 3 machine set up:
zoo.cfg file
server.1=192.168.33.10:2888:3888
server.2=192.168.33.11:2888:3888
server.3=192.168.33.12:2888:3888

echo "2" > /var/zookeeper/data/myid
echo "3" > /var/zookeeper/data/myid

Just to make sure that we have everything at this point, let's shut everything down and start it back up:
On each VM.
exit
vagrant halt

vagrant up
vagrant ssh

(Start Zookeeper and Kafka)
sudo $ZK_HOME/bin/zkServer.sh start
sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

Use commands to add topics and messages and read them. Have fun!