Saturday, November 22, 2014

Network & sharing in VirtualBox - Full tutorial

Disclaimer: This post was not made by me, I just wanted to "back up" essential to me parts here in case original went down. You can find the original at http://www.dedoimedo.com/computers/virtualbox-network-sharing.html

------------

This is the fourth article on VirtualBox management. Today, I'm going to teach you everything you need to know of VirtualBox networking and sharing.

I'm going to show you three different methods of configuring your virtual machines and three different ways of sharing data between the host machine and the virtual machine. After mastering this tutorial, you will know all there is to know about using VirtualBox with fun and confidence. Follow me.














 

 

Introduction


For more details, you should read the following articles. They will provide you with the necessary background to following today's material with ease and pleasure:

How to install VirtualBox Guest Additions - Tutorial

VirtualBox 3 is amazing!

VirtualBox 3 Compiz slideshow

DirectX in VirtualBox 3 - Pure joy is here

Likewise, you should read the first three installments of this series:

How to clone disks in VirtualBox - Tutorial

How to add hard disks in VirtualBox - Tutorial

How to expand/shrink disks in VirtualBox - Tutorial

Now, let us begin.

VirtualBox network options

For any of your installed virtual machines, click on Settings > Network. Here the fun begins. This is the default view. Any virtual machine can have up to four network adapters. You can enable them selectively as you see fit. Most people will require just one.

Adapter Type defines the virtualized hardware that VirtualBox will expose to your virtual machine. If you have a problem with one of the Adapter types, you can try another. PCnet-FAST III is the default selection.

You also have PCnet-FAST II for older machines and three types of Intel PRO/1000 cards, including two Server versions, which should be useful for people running VirtualBox in a production environment. For home users, the choice is rather transparent.


The most interesting part is Attached to: section. This category defines how your network adapter will interface with existing physical hardware. Different setups will result in markedly different results.


Network types


We have four options here: NAT (default), Bridged, Internal network, and Host Only. Of course, Not attached is also a type, but not one we can really use, per se.

Network Address Translation (NAT)

NAT means the virtual machines will have private IP addresses that are not routable from outside.

Example: Your host is 192.168.1.1. The VirtualBox NAT device will be marked as 10.0.2.1. Therefore, the virtual machines will be given any address in the 10.0.2.x range. Since there is nothing to route access to machines in the 10.0.2.x/24 subnet, they will be inaccessible from your host.


This setup is useful when you don't really care what IP addresses your guests have, each one to its own. However, it is not good if you require forwarding or if you need to expose services to the external world. Likewise, this setup is not good for sharing via network access.

Pluses: simplicity & seclusion.

Minuses: no route to virtual machines, no network sharing.

Bridged Adapter

Bridged Adapter means that any virtual machine running will try to obtain an IP address from the same source your currently active, default network address got its IP address. Hence the term bridged, as the two are connected.

Bridged

If you have more than one active network device, you can choose which one you want to bridge with VirtualBox. In our case, we will use the Wireless adapter wlan0.

Example: Your host has leased an address of 192.168.1.100 from the router. The virtual machine leases an address of 192.168.1.103 from the router. The two machines now share the same network and all standard rules apply. For all practical purposes, the virtual machine is another IP address on your LAN.


More closely, the host:



And the guest:



This setup cannot work if your device (switch, router, ISP, etc) does not permit you to lease more than one IP address. Therefore, computers with direct Internet access may not be able to use Bridged networking.

Pluses: Allows flexible management of the network with port forwarding and services enabled. Allows network sharing in the classic way.

Minuses: Might not work with direct Internet access (requires router), more difficult to understand for new users, exposes machines to network with possible security implications.

Host-only Adapter

Host-only Adapter is very interesting. It's very similar to Bridged Adapter, except that is uses a dedicated network device, called vboxnet0, to lease IP addresses.


Your host machine is the de-facto VirtualBox router, with the IP address of 192.168.56.1. The adapter is not in use if there are no virtual machines running with Host-only setup. However, once they come up, this adapter serves IP addresses to the virtual machines, creating an internal LAN, within your own network.

Example: Your host has the IP address of 192.168.56.1. Your virtual machine has the IP address of 192.168.56.101.


More closely, host:

Host

And the guest:


This is quite similar to what VMware Server does. VMware Server has its two virtual adapters called vmnet1 and vmnet8, which are used assign NAT and host-only IP addresses to guests. However, unlike the VirtualBox NAT adapter, VMware Server always bridges the default network device on your hosts and therefore you have direct network access to NAT-ed machines. You don't have this luxury on VirtualBox (yet).

But the addition of vboxnet0 in VirtualBox 3 has significantly simplified network usage in this phenomenal product. If you wish to recall the trouble I've had to deal with in earlier release of VirtualBox, do take a look at my VMGL tutorial. I had to manually configure everything. BTW, you can change the default IP address allocation, if you want.

Very importantly, please note that using the Host-only adapter does not mean your guests will have Internet access. In fact, they won't. vboxnet0 does not have a default gateway. To make vboxnet0 also serve queries outside the local network, you will have to configure it to use another adapter for that, enable forwarding and possible reconfigure your firewall rules. At the end, you will have achieved Bridged networking, so why bother?
Host-only Adapter is useful for creating private networks, where machines need access to one another, but not necessarily outside this subnet.

Pluses: Useful for noisy software testing, penetration testing. Allows classic network sharing via IP address.

Minuses: As difficult to understand as Bridged networking for new users, no Internet access in the virtual machines. May introduce a security risk to other machines on the private network.

Internal network

Internal network is not very interesting, in my opinion. It's similar to Host-only + NAT, except the networking takes place inside the virtual network of guest machines, without any access for the host, plus there is no real NAT. What you get is a private LAN for your guests only, without any access to the external world.



Tuesday, November 18, 2014

Getting Ambari up and running (with Vagrant)


I assume that you already have VirtualBox and Vagrant installed...

Set up


On guest machine, create a folder that would contain files for the VM
mkdir hadoop_ambari

Change to it and issue command to download the VM box and to add it to your library of VMs with specific name.
cd hadoop_ambari
vagrant box add hadoop_ambari https://github.com/2creatives/vagrant-centos/releases/download/v6.5.1/centos65-x86_64-20131205.box

Once download is complete, we can initialize Vagrant, which in turn would create Vagrantfile that acts as a configuration file. Various options are available to be specified: memory, ip, ports, etc.
vagrant init hadoop_ambari

Open Vagrantfile with vi editor and make following changes
config.vm.network :forwarded_port, guest: 8080, host: 8080
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provider "virtualbox" do |vb|  #
     # Use VBoxManage to customize the VM. For example to change memory:
     vb.customize ["modifyvm", :id, "--memory", "8192"]

end

Here we assigned static ip, opened port 8080 and made sure that we have 8GB of memory allocated to the machine.

Save the file and start vagrant
vagrant up

Once it starts, log in and change to root
vagrant ssh

inside of guest OS
sudo su
cd ~

Find out hostname of the machine
hostname

Edit /etc/hosts file to the following
vi /etc/hosts

192.168.33.11 <hostname>

(this ip was specified as static inside of your Vagrant file)

Install NTP service
yum install ntp

Install wget Utility
yum install wget

Turn on NTP service
chkconfig ntpd on
service ntpd start


Set up passwordless SSH
ssh-keygen
cd .ssh
cp id_rsa /vagrant
cat id_rsa.pub >> authorized_keys

Setup Ambari
wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.3.38/ambari.repo
cp ambari.repo /etc/yum.repos.d


Double check that repo was created
yum repolist

Install Ambari server
yum install ambari-server

Configure it (go with defaults)
ambari-server setup

Start Ambari Server
ambari-server start

Wait a little and you should be able to access your server at http://192.168.33.11:8080. Username and password: ambari/ambari. Follow the wizard which is self explanatory. In Install Options specify hostname of guest machine and then provide ssh private key by navigating to the hadoop_ambari folder that contains your Vagrantfile and id_rsa (remember when you copied your id_rsa file in guest OS to /vagrant folder?). In Customize Services, pick passwords for the services.

Wait for the installation to finish and enjoy your new set up! When browsing URLs inside of ambari, it by default would try to link to hostname and that won't work, so use the static ip instead. For example, MR2 JobHistory UI would be on http://192.168.33.11:19888.

Like always, comments? questions? just post!

Troubleshooting:

1.

If you can not get to the URLs, try to disable iptables
service iptables stop

Verify that curl is able to access the apache test page from inside the vm
vagrant ssh
curl -v localhost

If that doesn't work, then it's definitely not the port forwarding.

Lastly verify that the Host can access the page through curl
curl -v 'http://localhost:8080'

2.

Check if server is listening on 8080 and where it is binding
netstat -ntlp | grep 8080

Note that 127.0.0.1 is only accessible to the local machine, which for a guest machine means nothing.  Outside of the VM, it can't reach it! 0.0.0.0 is accessible from anywhere on the local network, which to a VM includes the host machine.

127.0.0.1 is normally the IP address assigned to the "loopback" or local-only interface. This is a "fake" network adapter that can only communicate within the same host. It's often used when you want a network-capable application to only serve clients on the same host. A process that is listening on 127.0.0.1 for connections will only receive local connections on that socket.

"localhost" is normally the hostname for the 127.0.0.1 IP address. It's usually set in /etc/hosts (or the Windows equivalent named "hosts" somewhere under %WINDIR%). You can use it just like any other hostname - try "ping localhost" to see how it resolves to 127.0.0.1.

0.0.0.0 has a couple of different meanings, but in this context, when a server is told to listen on 0.0.0.0 that means "listen on every available network interface". The loopback adapter with IP address 127.0.0.1 from the perspective of the server process looks just like any other network adapter on the machine, so a server told to listen on 0.0.0.0 will accept connections on that interface too.

Resources:

List of ports

  config.vm.network :forwarded_port, guest: 80, host: 42080, auto_correct: true #Apache http
  config.vm.network :forwarded_port, guest: 111, host: 42111, auto_correct: true #NFS portmap
  config.vm.network :forwarded_port, guest: 2223, host: 2223, auto_correct: true #Gateway node
  config.vm.network :forwarded_port, guest: 8000, host: 8000, auto_correct: true #Hue
  config.vm.network :forwarded_port, guest: 8020, host: 8020, auto_correct: true #Hdfs
  config.vm.network :forwarded_port, guest: 8042, host: 8042, auto_correct: true #NodeManager
  config.vm.network :forwarded_port, guest: 8050, host: 8050, auto_correct: true #Resource manager
  config.vm.network :forwarded_port, guest: 8080, host: 8080, auto_correct: true #Ambari
  config.vm.network :forwarded_port, guest: 8088, host: 8088, auto_correct: true #Yarn RM
  config.vm.network :forwarded_port, guest: 8443, host: 8443, auto_correct: true #Knox gateway
  config.vm.network :forwarded_port, guest: 8744, host: 8744, auto_correct: true #Storm UI
  config.vm.network :forwarded_port, guest: 8888, host: 8888, auto_correct: true #Tutorials
  config.vm.network :forwarded_port, guest: 10000, host: 10000, auto_correct: true #HiveServer2 thrift
  config.vm.network :forwarded_port, guest: 10001, host: 10001, auto_correct: true #HiveServer2 thrift http
  config.vm.network :forwarded_port, guest: 11000, host: 11000, auto_correct: true #Oozie
  config.vm.network :forwarded_port, guest: 15000, host: 15000, auto_correct: true #Falcon
  config.vm.network :forwarded_port, guest: 19888, host: 19888, auto_correct: true #Job history
  config.vm.network :forwarded_port, guest: 50070, host: 50070, auto_correct: true #WebHdfs
  config.vm.network :forwarded_port, guest: 50075, host: 50075, auto_correct: true #Datanode
  config.vm.network :forwarded_port, guest: 50111, host: 50111, auto_correct: true #WebHcat
  config.vm.network :forwarded_port, guest: 60080, host: 60080, auto_correct: true #WebHBase


References:

https://github.com/petro-rudenko/bigdata-toolbox/blob/master/Vagrantfile
http://serverfault.com/questions/513654/troubleshooting-why-1-vagrant-works-but-another-does-not
http://stackoverflow.com/questions/5984217/vagrants-port-forwarding-not-working
http://stackoverflow.com/questions/23840098/empty-reply-from-server-cant-connect-to-vagrant-vm-w-port-forwarding
http://stackoverflow.com/questions/20778771/what-is-the-difference-between-0-0-0-0-127-0-0-1-and-localhost


Thursday, November 13, 2014

How to enable backspace in vim on Mac OS

Create a file called ~/.vimrc and put the following lines in it.

set nocompatible
set backspace=indent,eol,start

ENJOY!