Logstash: Processing files from beginning

It was a while since I posted, so figured I would break the rust and write a small post.

I was recently involved in a project leveraging ELK (ElasticSearch, Logstash, Kibana). During the set up process, I had to experiment with different things and used a file with some test data. Well, after first successful try, my file was read in and Logstash "stopped" reading it...

What was wrong? Simple.

Logstash keeps track of files that it read and position in that file, so if you have a file with 5 lines, Logstash would read that file and mark that it read 5 lines in it. Next time it sees this file, it would check number of lines in it vs what it already processed. If no new lines were added file will be simply ignored. It is done so that same files wont be rescanned from the beginning each time Logstash sees them. It is good for production, but how can you rescan the file while testing? Simple!

First of all in your input block add

start_position => "beginning" 

and if you still experience problems,  run

rm ~/.sincedb*

before running logstash, in addition to start_position => beginning. This command would delete Logstash pointer data. Obviously this should be your last resort. :)

Why you should not open 42.zip.... or any zip file that you can't trust

42.zip(42.374 bytes zipped)

The file contains 16 zipped files, which again contains 16 zipped files, which again contains 16 zipped files, which again contains 16 zipped,
which again contains 16 zipped files, which contain 1 file, with the size of 4.3GB.

So, if you extract all files, you will most likely run out of space :-)

16 x 4294967295       = 68.719.476.720 (68GB)
16 x 68719476720      = 1.099.511.627.520 (1TB)
16 x 1099511627520    = 17.592.186.040.320 (17TB)
16 x 17592186040320   = 281.474.976.645.120 (281TB)
16 x 281474976645120  = 4.503.599.626.321.920 (4,5PB)

Notes from Lucas Carlson talk

6 Bad practices:
    1. Synchronicity
        3_tier systems are like sync information factory lines
        If one area chokes, rest chokes
    2. Dependency on Filesystems
        State data (sessions, uploaded files, etc) stored on hard drives is very hard to scale
    3. Heavy Server-Side Processing
        Generating all the HTML server-side made sense when client hardware was slow
        No REST/JSON/Javascript
        All the information from a request needs to be compiled before returning any data
    4. Expecting Services to Always be Available
        Designing for success is failure
        Cloud infrastructure has higher failure rates than dedicated hardware
        Disaster recovery can be slow and prone to errors
    5. Moving to the Cloud without a Plan
        Cloud migration is often thought of as simple a cost issue, not a technical one
        Higher failure rates in cloud infrastructure will break fragile applications
        Migrations without a good plan can cost a lot of unexpected time and money
    6. Lack of Redundancy
        All single points of failure are terrible monsters (DNS, Load Balancers, Network, etc)
        Not only choke points, but can take down otherwise robust system
        All your eggs in one basket
       
6 Good practices:
    1. Asynchronous Processes
        Small decoupled apps
        Communicate through a queue, REST APIs, or a message bus
        Each one should do one thing very well: simple and specialized
    2. Distributed Object Storage
        Memcached, Redis, MongoDB, CouchDB, etc
        Use instead of filesystems in legacy web applications (sessions, file uploads, etc)
        Consider replacing oc caching the largest and fastest growing relational database tables with object storage
    3. Micro-Services
        Leverage increased CPU capacity on browsers with client-side Javascript (AngularJS, Ember, Backbone)
        Simple and specialized REST APIs
            Java: Spring, Spark, Jersey
            .Net: WCF, OpenRasta
            Ruby: Sinatra
            Python: Flask, Bottle
            PHP: Slim
        Bonus: Power your mobile apps
    4. Architecting fo Failure
        Think about anti-fragile upfront
        Pro-actively stress your system and study how it fails (not just load testing, think of Netflix's chaos monkey)
        Make all failures an opportunity to eliminate bottlenecks, increase redundancy.
    5. Use cloud migration as an Opportunity to modernize architecture
        Dont half-do it
        Not all applications will do well in cloud environment
        Automation is vital in cloud environments where infrastructure isn't reliable
    6. Redundancy everywhere
        Audit every area of your application for redundancy
        2x or 3x redundancy is not enough
        Google's rule of thumb is for 5x redundancy
        Be like the Hydra, kill one head and grow tow in its spot

Netflix: Chaos Monkey - try to randomly break servers through out the system and day to see how your application would react.
                            Dont expect it to work  
                           
3 Conclusions
    1. Legacy web applications are fragile
    2. Think about anti-fragility (not scalability) up front
    3. Micro-services are anti-fragile future:
        Lightweight distributed
        Share-nothing systems built with APIs
       
Conclusion
    1. Automation is dominating the future landscape of web app architecture
    2. Micro-services are the future
        - Lightweight distributed
        - Share-nothing systems built with APIs
    3. Docker and Linux Containers are the new way to package and distribute your applications
   

Joys of Java 8... as was discovered during maven build that was trying to process wsdl file

Blah!

I was just trying to Clean&Build my work on a different machine and suddenly I had build failure! WTH!?!?

The error stated

Failed to read schema document 'xyz.xsd' because 'file' access is not allowed due to restriction set by the accessExternalSchema property

Never seen this one before and I was trying to figure out if I could not read checked out from git files for some reason. After few tweaks here and there, same issue. So I tried to google this error and I hit many Java 8 issues precessing wsdl, etc posts. Indeed my old machine was running Java 7, while new one was running 8!

On my Mac the solution was pretty easy

Go to 

cd /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/lib/

And create file jaxp.properties

And add this line to it

javax.xml.accessExternalSchema = all

Per http://stackoverflow.com/questions/23011547/webservice-client-generation-error-with-jdk8

Wow... oh well :)

Network & sharing in VirtualBox - Full tutorial

Disclaimer: This post was not made by me, I just wanted to "back up" essential to me parts here in case original went down. You can find the original at http://www.dedoimedo.com/computers/virtualbox-network-sharing.html

------------

This is the fourth article on VirtualBox management. Today, I'm going to teach you everything you need to know of VirtualBox networking and sharing.

I'm going to show you three different methods of configuring your virtual machines and three different ways of sharing data between the host machine and the virtual machine. After mastering this tutorial, you will know all there is to know about using VirtualBox with fun and confidence. Follow me.

 

 

Introduction

For more details, you should read the following articles. They will provide you with the necessary background to following today's material with ease and pleasure:

How to install VirtualBox Guest Additions - Tutorial

VirtualBox 3 is amazing!

VirtualBox 3 Compiz slideshow

DirectX in VirtualBox 3 - Pure joy is here

Likewise, you should read the first three installments of this series:

How to clone disks in VirtualBox - Tutorial

How to add hard disks in VirtualBox - Tutorial

How to expand/shrink disks in VirtualBox - Tutorial

Now, let us begin.

VirtualBox network options

For any of your installed virtual machines, click on Settings > Network. Here the fun begins. This is the default view. Any virtual machine can have up to four network adapters. You can enable them selectively as you see fit. Most people will require just one.

Adapter Type defines the virtualized hardware that VirtualBox will expose to your virtual machine. If you have a problem with one of the Adapter types, you can try another. PCnet-FAST III is the default selection.

You also have PCnet-FAST II for older machines and three types of Intel PRO/1000 cards, including two Server versions, which should be useful for people running VirtualBox in a production environment. For home users, the choice is rather transparent.

The most interesting part is Attached to: section. This category defines how your network adapter will interface with existing physical hardware. Different setups will result in markedly different results.

Network types

We have four options here: NAT (default), Bridged, Internal network, and Host Only. Of course, Not attached is also a type, but not one we can really use, per se.

Network Address Translation (NAT)

NAT means the virtual machines will have private IP addresses that are not routable from outside.

Example: Your host is 192.168.1.1. The VirtualBox NAT device will be marked as 10.0.2.1. Therefore, the virtual machines will be given any address in the 10.0.2.x range. Since there is nothing to route access to machines in the 10.0.2.x/24 subnet, they will be inaccessible from your host.

This setup is useful when you don't really care what IP addresses your guests have, each one to its own. However, it is not good if you require forwarding or if you need to expose services to the external world. Likewise, this setup is not good for sharing via network access.

Pluses: simplicity & seclusion.

Minuses: no route to virtual machines, no network sharing.

Bridged Adapter

Bridged Adapter means that any virtual machine running will try to obtain an IP address from the same source your currently active, default network address got its IP address. Hence the term bridged, as the two are connected.

Bridged

If you have more than one active network device, you can choose which one you want to bridge with VirtualBox. In our case, we will use the Wireless adapter wlan0.

Example: Your host has leased an address of 192.168.1.100 from the router. The virtual machine leases an address of 192.168.1.103 from the router. The two machines now share the same network and all standard rules apply. For all practical purposes, the virtual machine is another IP address on your LAN.

More closely, the host:

And the guest:

This setup cannot work if your device (switch, router, ISP, etc) does not permit you to lease more than one IP address. Therefore, computers with direct Internet access may not be able to use Bridged networking.

Pluses: Allows flexible management of the network with port forwarding and services enabled. Allows network sharing in the classic way.

Minuses: Might not work with direct Internet access (requires router), more difficult to understand for new users, exposes machines to network with possible security implications.

Host-only Adapter

Host-only Adapter is very interesting. It's very similar to Bridged Adapter, except that is uses a dedicated network device, called vboxnet0, to lease IP addresses.

Your host machine is the de-facto VirtualBox router, with the IP address of 192.168.56.1. The adapter is not in use if there are no virtual machines running with Host-only setup. However, once they come up, this adapter serves IP addresses to the virtual machines, creating an internal LAN, within your own network.

Example: Your host has the IP address of 192.168.56.1. Your virtual machine has the IP address of 192.168.56.101.

More closely, host:

Host

And the guest:

This is quite similar to what VMware Server does. VMware Server has its two virtual adapters called vmnet1 and vmnet8, which are used assign NAT and host-only IP addresses to guests. However, unlike the VirtualBox NAT adapter, VMware Server always bridges the default network device on your hosts and therefore you have direct network access to NAT-ed machines. You don't have this luxury on VirtualBox (yet).

But the addition of vboxnet0 in VirtualBox 3 has significantly simplified network usage in this phenomenal product. If you wish to recall the trouble I've had to deal with in earlier release of VirtualBox, do take a look at my VMGL tutorial. I had to manually configure everything. BTW, you can change the default IP address allocation, if you want.

Very importantly, please note that using the Host-only adapter does not mean your guests will have Internet access. In fact, they won't. vboxnet0 does not have a default gateway. To make vboxnet0 also serve queries outside the local network, you will have to configure it to use another adapter for that, enable forwarding and possible reconfigure your firewall rules. At the end, you will have achieved Bridged networking, so why bother?
Host-only Adapter is useful for creating private networks, where machines need access to one another, but not necessarily outside this subnet.

Pluses: Useful for noisy software testing, penetration testing. Allows classic network sharing via IP address.

Minuses: As difficult to understand as Bridged networking for new users, no Internet access in the virtual machines. May introduce a security risk to other machines on the private network.

Internal network

Internal network is not very interesting, in my opinion. It's similar to Host-only + NAT, except the networking takes place inside the virtual network of guest machines, without any access for the host, plus there is no real NAT. What you get is a private LAN for your guests only, without any access to the external world.

Getting Ambari up and running (with Vagrant)

I assume that you already have VirtualBox and Vagrant installed...

Set up

On guest machine, create a folder that would contain files for the VM
mkdir hadoop_ambari

Change to it and issue command to download the VM box and to add it to your library of VMs with specific name.
cd hadoop_ambari
vagrant box add hadoop_ambari https://github.com/2creatives/vagrant-centos/releases/download/v6.5.1/centos65-x86_64-20131205.box

Once download is complete, we can initialize Vagrant, which in turn would create Vagrantfile that acts as a configuration file. Various options are available to be specified: memory, ip, ports, etc.
vagrant init hadoop_ambari

Open Vagrantfile with vi editor and make following changes
config.vm.network :forwarded_port, guest: 8080, host: 8080
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provider "virtualbox" do |vb|  #
     # Use VBoxManage to customize the VM. For example to change memory:
     vb.customize ["modifyvm", :id, "--memory", "8192"]
end

Here we assigned static ip, opened port 8080 and made sure that we have 8GB of memory allocated to the machine.

Save the file and start vagrant
vagrant up

Once it starts, log in and change to root
vagrant ssh

inside of guest OS
sudo su
cd ~

Find out hostname of the machine
hostname

Edit /etc/hosts file to the following
vi /etc/hosts

192.168.33.11 <hostname>

(this ip was specified as static inside of your Vagrant file)

Install NTP service
yum install ntp

Install wget Utility
yum install wget

Turn on NTP service
chkconfig ntpd on
service ntpd start

Set up passwordless SSH
ssh-keygen
cd .ssh
cp id_rsa /vagrant
cat id_rsa.pub >> authorized_keys

Setup Ambari
wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.3.38/ambari.repo
cp ambari.repo /etc/yum.repos.d

Double check that repo was created
yum repolist

Install Ambari server
yum install ambari-server

Configure it (go with defaults)
ambari-server setup

Start Ambari Server
ambari-server start

Wait a little and you should be able to access your server at http://192.168.33.11:8080. Username and password: ambari/ambari. Follow the wizard which is self explanatory. In Install Options specify hostname of guest machine and then provide ssh private key by navigating to the hadoop_ambari folder that contains your Vagrantfile and id_rsa (remember when you copied your id_rsa file in guest OS to /vagrant folder?). In Customize Services, pick passwords for the services.

Wait for the installation to finish and enjoy your new set up! When browsing URLs inside of ambari, it by default would try to link to hostname and that won't work, so use the static ip instead. For example, MR2 JobHistory UI would be on http://192.168.33.11:19888.

Like always, comments? questions? just post!

Troubleshooting:

1.

If you can not get to the URLs, try to disable iptables
service iptables stop

Verify that curl is able to access the apache test page from inside the vm
vagrant ssh
curl -v localhost
If that doesn't work, then it's definitely not the port forwarding.

Lastly verify that the Host can access the page through curl
curl -v 'http://localhost:8080'

2.

Check if server is listening on 8080 and where it is binding
netstat -ntlp | grep 8080

Note that 127.0.0.1 is only accessible to the local machine, which for a guest machine means nothing.  Outside of the VM, it can't reach it! 0.0.0.0 is accessible from anywhere on the local network, which to a VM includes the host machine.

127.0.0.1 is normally the IP address assigned to the "loopback" or local-only interface. This is a "fake" network adapter that can only communicate within the same host. It's often used when you want a network-capable application to only serve clients on the same host. A process that is listening on 127.0.0.1 for connections will only receive local connections on that socket.

"localhost" is normally the hostname for the 127.0.0.1 IP address. It's usually set in /etc/hosts (or the Windows equivalent named "hosts" somewhere under %WINDIR%). You can use it just like any other hostname - try "ping localhost" to see how it resolves to 127.0.0.1.

0.0.0.0 has a couple of different meanings, but in this context, when a server is told to listen on 0.0.0.0 that means "listen on every available network interface". The loopback adapter with IP address 127.0.0.1 from the perspective of the server process looks just like any other network adapter on the machine, so a server told to listen on 0.0.0.0 will accept connections on that interface too.

Resources:

List of ports

  config.vm.network :forwarded_port, guest: 80, host: 42080, auto_correct: true #Apache http
  config.vm.network :forwarded_port, guest: 111, host: 42111, auto_correct: true #NFS portmap
  config.vm.network :forwarded_port, guest: 2223, host: 2223, auto_correct: true #Gateway node
  config.vm.network :forwarded_port, guest: 8000, host: 8000, auto_correct: true #Hue
  config.vm.network :forwarded_port, guest: 8020, host: 8020, auto_correct: true #Hdfs
  config.vm.network :forwarded_port, guest: 8042, host: 8042, auto_correct: true #NodeManager
  config.vm.network :forwarded_port, guest: 8050, host: 8050, auto_correct: true #Resource manager
  config.vm.network :forwarded_port, guest: 8080, host: 8080, auto_correct: true #Ambari
  config.vm.network :forwarded_port, guest: 8088, host: 8088, auto_correct: true #Yarn RM
  config.vm.network :forwarded_port, guest: 8443, host: 8443, auto_correct: true #Knox gateway
  config.vm.network :forwarded_port, guest: 8744, host: 8744, auto_correct: true #Storm UI
  config.vm.network :forwarded_port, guest: 8888, host: 8888, auto_correct: true #Tutorials
  config.vm.network :forwarded_port, guest: 10000, host: 10000, auto_correct: true #HiveServer2 thrift
  config.vm.network :forwarded_port, guest: 10001, host: 10001, auto_correct: true #HiveServer2 thrift http
  config.vm.network :forwarded_port, guest: 11000, host: 11000, auto_correct: true #Oozie
  config.vm.network :forwarded_port, guest: 15000, host: 15000, auto_correct: true #Falcon
  config.vm.network :forwarded_port, guest: 19888, host: 19888, auto_correct: true #Job history
  config.vm.network :forwarded_port, guest: 50070, host: 50070, auto_correct: true #WebHdfs
  config.vm.network :forwarded_port, guest: 50075, host: 50075, auto_correct: true #Datanode
  config.vm.network :forwarded_port, guest: 50111, host: 50111, auto_correct: true #WebHcat
  config.vm.network :forwarded_port, guest: 60080, host: 60080, auto_correct: true #WebHBase

References:

https://github.com/petro-rudenko/bigdata-toolbox/blob/master/Vagrantfile
http://serverfault.com/questions/513654/troubleshooting-why-1-vagrant-works-but-another-does-not
http://stackoverflow.com/questions/5984217/vagrants-port-forwarding-not-working
http://stackoverflow.com/questions/23840098/empty-reply-from-server-cant-connect-to-vagrant-vm-w-port-forwarding
http://stackoverflow.com/questions/20778771/what-is-the-difference-between-0-0-0-0-127-0-0-1-and-localhost

How to enable backspace in vim on Mac OS

Create a file called ~/.vimrc and put the following lines in it.

set nocompatible
set backspace=indent,eol,start

ENJOY!