<resources>
<j2se version="1.7+" />
...
...
<property name="prop1" value="value1"/>
<property name="prop2" value="value2"/>
</resources>and then call it in your code, like so
System.getProperty("prop1");
Monday, April 28, 2014
How to pass parameters to your webstart?
Add properties in the JNLP file itself, like so
<resources>
and then call it in your code, like so
System.getProperty("prop1");
<resources>
and then call it in your code, like so
System.getProperty("prop1");
Where am I?
While calling something like
FileInputStream stream = new FileInputStream("myfile.properties");
you might see Filenotfound exception... But it used to work before, you might say. Well, it probably did before you called this method from a different directory. Use following statement to get user's current working directory.
System.getProperty("user.dir")
FileInputStream stream = new FileInputStream("myfile.properties");
you might see Filenotfound exception... But it used to work before, you might say. Well, it probably did before you called this method from a different directory. Use following statement to get user's current working directory.
System.getProperty("user.dir")
Sunday, April 13, 2014
Connecting to the Accumulo using Java
Now that our Accumulo is set up, let’s connect to it through Java App so we can create our first table and write some data into it.
(I am going to assume that you have maven installed on your machine… If not, see this)
Create simple maven project.
mvn archetype:generate -DgroupId=org.mytest.accexample -DartifactId=simple -Dpackage=org.mytest.accexample -Dversion=1.0-SNAPSHOT
Accept all defaults.
Since we will be connecting to Accumulo, it comes to no surprise that we would need jars to support Accumulo, Zookeeper and Hadoop. So add this to your pom.xml file
<dependency>
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-core</artifactId>
<version>1.4.4</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5</version>
</dependency>
Now it is time to write a program to connect to our Accumulo instance and put some records in!
package org.mytest.accexample;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.client.admin.TableOperations;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.ColumnVisibility;
import org.apache.hadoop.io.Text;
public class App {
public static void main(String[] args) throws AccumuloException, AccumuloSecurityException,TableNotFoundException, TableExistsException {
// Constants
String instanceName = "default";
String zooServers = ""; // Provide list of zookeeper server here. In our case, we had just one so localhost:2181 should do
String userName = ""; // Provide username
String password = ""; // Provide password
// Connect
Instance inst = new ZooKeeperInstance(instanceName,zooServers);
Connector conn = inst.getConnector(userName, password);
// Let’s create our new table
String tableName = "myTable";
TableOperations ops = conn.tableOperations();
if (ops.exists(tableName)) {
ops.delete(tableName);
}
ops.create(tableName);
// Use batch writer to write demo data
BatchWriter bw = conn.createBatchWriter(tableName,1000000, 60000, 2);
// set values
Text rowID = new Text("row1");
Text colFam = new Text("colFam");
Text colQual = new Text("colQual");
// set value
Value value = new Value("some-value".getBytes());
// create new mutation and add rowID, colFam, colQual, and value
Mutation mutation = new Mutation(rowID);
mutation.put(colFam, colQual, value);
// add the mutation to the batch writer
bw.addMutation(mutation);
// close the batch writer
bw.close();
}
}
Run it… Congratulations you just wrote your first record into Accumulo! Now let’s view it in the table just to make sure it is indeed there.
Let’s take a different approach and access Accumulo through bash shell.
$ACCUMULO_HOME/bin/accumulo shell -u [username]
Then provide your password.
Type ‘tables’ to see a list of available tables. You should see myTable in the list.
Type ‘table myTable’ to access it. Now execute simple scan to view your records.
root@default myTable> table myTable
root@default myTable> scan
row1 colFam:colQual [] some-value
Success!
Now, what’s that security level stuff in Accumulo everyone is talking about? Well, it is a way to ensure that only rows that fit your authorization will be shown to you. That’s it in the nutshell, for more in details explanation please refer to this.
Let’s add this to our Java App Accumulo writer. Add ColumnVisibility variable and add it to the mutation object. Here is the complete example with changes highlighted in red:
package org.mytest.accexample;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.client.admin.TableOperations;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.ColumnVisibility;
import org.apache.hadoop.io.Text;
public class App {
public static void main(String[] args) throws AccumuloException, AccumuloSecurityException,TableNotFoundException, TableExistsException {
// Constants
String instanceName = "default";
String zooServers = "";
String userName = "";
String password = "";
// Connect
Instance inst = new ZooKeeperInstance(instanceName,zooServers);
Connector conn = inst.getConnector(userName, password);
String tableName = "myTable";
TableOperations ops = conn.tableOperations();
if (ops.exists(tableName)) {
ops.delete(tableName);
}
ops.create(tableName);
// Use batch writer to write demo data
BatchWriter bw = conn.createBatchWriter(tableName,1000000, 60000, 2);
// set values
Text rowID = new Text("row1");
Text colFam = new Text("colFam");
Text colQual = new Text("colQual");
// set visibility
ColumnVisibility colVis = new ColumnVisibility("public");
// set value
Value value = new Value("some-value".getBytes());
// create new mutation and add rowID, colFam, colQual, and value
Mutation mutation = new Mutation(rowID);
mutation.put(colFam, colQual, colVis, value);
// add the mutation to the batch writer
bw.addMutation(mutation);
// close the batch writer
bw.close();
}
}
Run the example again… Now log into Accumulo shell and scan the myTable. You won’t see anything! What?
Well, you just wrote a record that would be shown only to users with authorization to view ‘public’ rows. What can you view right now for your user? Run ‘getauths’ command in the shell to find out.
‘public’ auth was probably not in the list, hence inability to see the freshly inserted record.
Let’s add ‘public’ to the list of authorizations. Run following command from the bash
root@default myTable> setauths -s public
Now run the scan and you should be able to see the row. Notice ‘public’ in square brackets
root@default myTable> scan
row1 colFam:colQual [public] some-value
In summary, we were able to write a java client that would communicate with Accumulo instance and would insert a record. We used Accumulo bash shell to view the record. We also briefly touched on cell level security.
Reference:
1. Accumulo Shell Command Guide.
2. Accumulo Visibility.
(I am going to assume that you have maven installed on your machine… If not, see this)
Create simple maven project.
mvn archetype:generate -DgroupId=org.mytest.accexample -DartifactId=simple -Dpackage=org.mytest.accexample -Dversion=1.0-SNAPSHOT
Accept all defaults.
Since we will be connecting to Accumulo, it comes to no surprise that we would need jars to support Accumulo, Zookeeper and Hadoop. So add this to your pom.xml file
<dependency>
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-core</artifactId>
<version>1.4.4</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5</version>
</dependency>
Now it is time to write a program to connect to our Accumulo instance and put some records in!
package org.mytest.accexample;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.client.admin.TableOperations;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.ColumnVisibility;
import org.apache.hadoop.io.Text;
public class App {
public static void main(String[] args) throws AccumuloException, AccumuloSecurityException,TableNotFoundException, TableExistsException {
// Constants
String instanceName = "default";
String zooServers = ""; // Provide list of zookeeper server here. In our case, we had just one so localhost:2181 should do
String userName = ""; // Provide username
String password = ""; // Provide password
// Connect
Instance inst = new ZooKeeperInstance(instanceName,zooServers);
Connector conn = inst.getConnector(userName, password);
// Let’s create our new table
String tableName = "myTable";
TableOperations ops = conn.tableOperations();
if (ops.exists(tableName)) {
ops.delete(tableName);
}
ops.create(tableName);
// Use batch writer to write demo data
BatchWriter bw = conn.createBatchWriter(tableName,1000000, 60000, 2);
// set values
Text rowID = new Text("row1");
Text colFam = new Text("colFam");
Text colQual = new Text("colQual");
// set value
Value value = new Value("some-value".getBytes());
// create new mutation and add rowID, colFam, colQual, and value
Mutation mutation = new Mutation(rowID);
mutation.put(colFam, colQual, value);
// add the mutation to the batch writer
bw.addMutation(mutation);
// close the batch writer
bw.close();
}
}
Run it… Congratulations you just wrote your first record into Accumulo! Now let’s view it in the table just to make sure it is indeed there.
Let’s take a different approach and access Accumulo through bash shell.
$ACCUMULO_HOME/bin/accumulo shell -u [username]
Then provide your password.
Type ‘tables’ to see a list of available tables. You should see myTable in the list.
Type ‘table myTable’ to access it. Now execute simple scan to view your records.
root@default myTable> table myTable
root@default myTable> scan
row1 colFam:colQual [] some-value
Success!
Now, what’s that security level stuff in Accumulo everyone is talking about? Well, it is a way to ensure that only rows that fit your authorization will be shown to you. That’s it in the nutshell, for more in details explanation please refer to this.
Let’s add this to our Java App Accumulo writer. Add ColumnVisibility variable and add it to the mutation object. Here is the complete example with changes highlighted in red:
package org.mytest.accexample;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.client.admin.TableOperations;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.ColumnVisibility;
import org.apache.hadoop.io.Text;
public class App {
public static void main(String[] args) throws AccumuloException, AccumuloSecurityException,TableNotFoundException, TableExistsException {
// Constants
String instanceName = "default";
String zooServers = "";
String userName = "";
String password = "";
// Connect
Instance inst = new ZooKeeperInstance(instanceName,zooServers);
Connector conn = inst.getConnector(userName, password);
String tableName = "myTable";
TableOperations ops = conn.tableOperations();
if (ops.exists(tableName)) {
ops.delete(tableName);
}
ops.create(tableName);
// Use batch writer to write demo data
BatchWriter bw = conn.createBatchWriter(tableName,1000000, 60000, 2);
// set values
Text rowID = new Text("row1");
Text colFam = new Text("colFam");
Text colQual = new Text("colQual");
// set visibility
ColumnVisibility colVis = new ColumnVisibility("public");
// set value
Value value = new Value("some-value".getBytes());
// create new mutation and add rowID, colFam, colQual, and value
Mutation mutation = new Mutation(rowID);
mutation.put(colFam, colQual, colVis, value);
// add the mutation to the batch writer
bw.addMutation(mutation);
// close the batch writer
bw.close();
}
}
Run the example again… Now log into Accumulo shell and scan the myTable. You won’t see anything! What?
Well, you just wrote a record that would be shown only to users with authorization to view ‘public’ rows. What can you view right now for your user? Run ‘getauths’ command in the shell to find out.
‘public’ auth was probably not in the list, hence inability to see the freshly inserted record.
Let’s add ‘public’ to the list of authorizations. Run following command from the bash
root@default myTable> setauths -s public
Now run the scan and you should be able to see the row. Notice ‘public’ in square brackets
root@default myTable> scan
row1 colFam:colQual [public] some-value
In summary, we were able to write a java client that would communicate with Accumulo instance and would insert a record. We used Accumulo bash shell to view the record. We also briefly touched on cell level security.
Reference:
1. Accumulo Shell Command Guide.
2. Accumulo Visibility.
Thursday, April 10, 2014
Run your jar as a server on Linux
Recently, I was experimenting with Apache Thrift. Then I decided to upload my server jar file onto RHEL server and connect to it with my client from my machine.
The set up was pretty simple. I generated jar file with NetBeans. This gave me actual jar file and a lib folder. I uploaded them to RHEL server and ran it
java -cp AdditionThriftSample.jar com.blogspot.alkrinker.samples.thrift.server.MyServer
Saw
Starting the simple server on 9091...
and I was able to connect to it from my Thrift client.
Now... I wanted to go a step further and make my thrift server to be a service. Here is what I did:
Create the start and the stop script of your application. Make sure to chmod them so they can be executed. Put it on some directory, in our example is:
Sample of the stop script:
Now make it a service! Create the following script (
Put the script to start with the system (using SysV). Just run the following command (as root):
The set up was pretty simple. I generated jar file with NetBeans. This gave me actual jar file and a lib folder. I uploaded them to RHEL server and ran it
java -cp AdditionThriftSample.jar com.blogspot.alkrinker.samples.thrift.server.MyServer
Saw
Starting the simple server on 9091...
and I was able to connect to it from my Thrift client.
Now... I wanted to go a step further and make my thrift server to be a service. Here is what I did:
Create the start and the stop script of your application. Make sure to chmod them so they can be executed. Put it on some directory, in our example is:
- Start Script:
/usr/local/bin/AdditionThriftSample-start.sh
- Stop Script:
/usr/local/bin/AdditionThriftSample-stop.sh
AdditionThriftSample-start.sh content: #!/bin/bash
java -cp AdditionThriftSample.jar com.blogspot.alkrinker.samples.thrift.server.MyServer Sample of the stop script:
#!/bin/bash
# Grabs and kill a process from the pidlist that has the word myapp
pid=`ps aux | grep AdditionThriftSample | awk '{print $2}'`
kill -9 $pidNow make it a service! Create the following script (
thriftserver) and put it on /etc/init.d./etc/init.d/thriftserver content:#!/bin/bash
# thriftserver
#
case $1 in
start)
/bin/bash /usr/local/bin/AdditionThriftSample-start.sh
;;
stop)
/bin/bash /usr/local/bin/AdditionThriftSample-stop.sh
;;
restart)
/bin/bash /usr/local/bin/AdditionThriftSample-stop.sh
/bin/bash /usr/local/bin/AdditionThriftSample-start.sh
;;
esac
exit 0Put the script to start with the system (using SysV). Just run the following command (as root):
update-rc.d thriftserver defaults Credits:Monday, April 7, 2014
Working Java REST Client Example to access CAS REST API
Ever wondered how to expose REST API in your CAS server? Well, the documentation is there, but I failed to find step-by-step guide... I ended up writing one and sharing it on stackoverflow website. Here is the link: Working Java REST Client Example to access CAS REST API . Enjoy and feel free to ask questions as always.
Wednesday, March 19, 2014
Installing Accumulo
We built Accumulo from the source code in our previous post, let’s now use it to install Accumulo locally.
First copy it over, extract it and then change permissions
[root@localhost local]# cp /opt/accumulo/source/accumulo/assemble/target/accumulo-1.6.0-SNAPSHOT-bin.tar.gz /usr/local/
[root@localhost local]# tar xzf accumulo-1.6.0-SNAPSHOT-bin.tar.gz
[root@localhost local]# mv accumulo-1.6.0-SNAPSHOT/ accumulo
[root@localhost local]# chown -R hduser:hadoop accumulo
Since we are going to be using Accumulo in our development environment mainly to fix bugs or to add features, we are going to use 512MB instance of Accumulo that is found inside of /accumulo/conf/examples. Other samples include 1GB, 2GB, and 3GB, which are overkill in our case.
Now, copy it over as hduser into your main conf directory
[root@localhost ~]# su – hduser
[hduser@localhost examples]$ cp /usr/local/accumulo/conf/examples/512MB/standalone/* /usr/local/accumulo/conf
[hduser@localhost conf]$ ll
total 64
-rwxr-xr-x. 1 hduser hduser 3101 Mar 3 23:48 accumulo-env.sh
-rw-r--r--. 1 hduser hduser 2180 Mar 3 23:48 accumulo-metrics.xml
-rw-r--r--. 1 hduser hadoop 7995 Mar 3 21:54 accumulo.policy.example
-rw-r--r--. 1 hduser hduser 4240 Mar 3 23:48 accumulo-site.xml
-rw-r--r--. 1 hduser hduser 1673 Mar 3 23:48 auditLog.xml
drwxr-xr-x. 8 hduser hadoop 4096 Mar 3 21:54 examples
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 gc
-rw-r--r--. 1 hduser hduser 3613 Mar 3 23:48 generic_logger.xml
-rw-r--r--. 1 hduser hduser 1713 Mar 3 23:48 log4j.properties
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 masters
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 monitor
-rw-r--r--. 1 hduser hduser 2924 Mar 3 23:48 monitor_logger.xml
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 slaves
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 tracers
Now that we have all configuration files in the right place, go ahead and specify location of your JAVA_HOME, ZOOKEEPER_HOME and HADOOP_HOME in accumulo-env.sh file located in /usr/local/accumulo/conf.
Before starting Accumulo for the very first time, you would have to run initialization script to create HDFS directory structure and ZooKeeper settings.
[hduser@localhost ~]$ /usr/local/accumulo/bin/accumulo init
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2014-03-04 00:40:01,555 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-03-04 00:40:03,065 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:40:03,065 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:40:03,066 [init.Initialize] INFO : Hadoop Filesystem is hdfs://localhost:9000
2014-03-04 00:40:03,067 [init.Initialize] INFO : Accumulo data dirs are [hdfs://localhost:9000/accumulo]
2014-03-04 00:40:03,067 [init.Initialize] INFO : Zookeeper server is localhost:2181
2014-03-04 00:40:03,067 [init.Initialize] INFO : Checking if Zookeeper is available. If this hangs, then you need to make sure zookeeper is running
Warning!!! Your instance secret is still set to the default, this is not secure. We highly recommend you change it.
You can change the instance secret in accumulo by using:
bin/accumulo org.apache.accumulo.server.util.ChangeSecret oldPassword newPassword.
You will also need to edit your secret in your configuration file by adding the property instance.secret to your conf/accumulo-site.xml. Without this accumulo will not operate correctlyInstance name : accumulo-demo
Enter initial password for root (this may not be applicable for your security setup): ***********
Confirm initial password for root: ***********
2014-03-04 00:41:11,785 [Configuration.deprecation] INFO : dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2014-03-04 00:41:12,239 [Configuration.deprecation] INFO : dfs.block.size is deprecated. Instead, use dfs.blocksize
2014-03-04 00:41:15,884 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKAuthorizor
2014-03-04 00:41:15,887 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKAuthenticator
2014-03-04 00:41:15,890 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKPermHandler
Now you are ready to start accumulo!
[hduser@localhost ~]$ /usr/local/accumulo/bin/start-all.sh
WARN : Using Zookeeper /usr/local/zookeeper/zookeeper-3.4.6. Use version 3.3.0 or greater to avoid zookeeper deadlock bug.
Starting monitor on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting tablet servers .... done
Starting tablet server on localhost
WARN : Max files open on localhost is 1024, recommend 65536
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2014-03-04 00:42:29,070 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-03-04 00:42:31,922 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:42:31,922 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:42:31,934 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-03-04 00:42:33,444 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
2014-03-04 00:42:33,654 [server.Accumulo] INFO : Connected to HDFS
Starting master on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting garbage collector on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting tracer on localhost
WARN : Max files open on localhost is 1024, recommend 65536
To confirm that it is up and running, view the Accumulo Overview page at: http://localhost:50095
First copy it over, extract it and then change permissions
[root@localhost local]# cp /opt/accumulo/source/accumulo/assemble/target/accumulo-1.6.0-SNAPSHOT-bin.tar.gz /usr/local/
[root@localhost local]# tar xzf accumulo-1.6.0-SNAPSHOT-bin.tar.gz
[root@localhost local]# mv accumulo-1.6.0-SNAPSHOT/ accumulo
[root@localhost local]# chown -R hduser:hadoop accumulo
Since we are going to be using Accumulo in our development environment mainly to fix bugs or to add features, we are going to use 512MB instance of Accumulo that is found inside of /accumulo/conf/examples. Other samples include 1GB, 2GB, and 3GB, which are overkill in our case.
Now, copy it over as hduser into your main conf directory
[root@localhost ~]# su – hduser
[hduser@localhost examples]$ cp /usr/local/accumulo/conf/examples/512MB/standalone/* /usr/local/accumulo/conf
[hduser@localhost conf]$ ll
total 64
-rwxr-xr-x. 1 hduser hduser 3101 Mar 3 23:48 accumulo-env.sh
-rw-r--r--. 1 hduser hduser 2180 Mar 3 23:48 accumulo-metrics.xml
-rw-r--r--. 1 hduser hadoop 7995 Mar 3 21:54 accumulo.policy.example
-rw-r--r--. 1 hduser hduser 4240 Mar 3 23:48 accumulo-site.xml
-rw-r--r--. 1 hduser hduser 1673 Mar 3 23:48 auditLog.xml
drwxr-xr-x. 8 hduser hadoop 4096 Mar 3 21:54 examples
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 gc
-rw-r--r--. 1 hduser hduser 3613 Mar 3 23:48 generic_logger.xml
-rw-r--r--. 1 hduser hduser 1713 Mar 3 23:48 log4j.properties
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 masters
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 monitor
-rw-r--r--. 1 hduser hduser 2924 Mar 3 23:48 monitor_logger.xml
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 slaves
-rw-r--r--. 1 hduser hduser 792 Mar 3 23:48 tracers
Now that we have all configuration files in the right place, go ahead and specify location of your JAVA_HOME, ZOOKEEPER_HOME and HADOOP_HOME in accumulo-env.sh file located in /usr/local/accumulo/conf.
Before starting Accumulo for the very first time, you would have to run initialization script to create HDFS directory structure and ZooKeeper settings.
[hduser@localhost ~]$ /usr/local/accumulo/bin/accumulo init
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2014-03-04 00:40:01,555 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-03-04 00:40:03,065 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:40:03,065 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:40:03,066 [init.Initialize] INFO : Hadoop Filesystem is hdfs://localhost:9000
2014-03-04 00:40:03,067 [init.Initialize] INFO : Accumulo data dirs are [hdfs://localhost:9000/accumulo]
2014-03-04 00:40:03,067 [init.Initialize] INFO : Zookeeper server is localhost:2181
2014-03-04 00:40:03,067 [init.Initialize] INFO : Checking if Zookeeper is available. If this hangs, then you need to make sure zookeeper is running
Warning!!! Your instance secret is still set to the default, this is not secure. We highly recommend you change it.
You can change the instance secret in accumulo by using:
bin/accumulo org.apache.accumulo.server.util.ChangeSecret oldPassword newPassword.
You will also need to edit your secret in your configuration file by adding the property instance.secret to your conf/accumulo-site.xml. Without this accumulo will not operate correctlyInstance name : accumulo-demo
Enter initial password for root (this may not be applicable for your security setup): ***********
Confirm initial password for root: ***********
2014-03-04 00:41:11,785 [Configuration.deprecation] INFO : dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2014-03-04 00:41:12,239 [Configuration.deprecation] INFO : dfs.block.size is deprecated. Instead, use dfs.blocksize
2014-03-04 00:41:15,884 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKAuthorizor
2014-03-04 00:41:15,887 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKAuthenticator
2014-03-04 00:41:15,890 [conf.AccumuloConfiguration] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKPermHandler
Now you are ready to start accumulo!
[hduser@localhost ~]$ /usr/local/accumulo/bin/start-all.sh
WARN : Using Zookeeper /usr/local/zookeeper/zookeeper-3.4.6. Use version 3.3.0 or greater to avoid zookeeper deadlock bug.
Starting monitor on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting tablet servers .... done
Starting tablet server on localhost
WARN : Max files open on localhost is 1024, recommend 65536
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2014-03-04 00:42:29,070 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-03-04 00:42:31,922 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:42:31,922 [fs.VolumeManagerImpl] WARN : dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is possible on system reset or power loss
2014-03-04 00:42:31,934 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-03-04 00:42:33,444 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
2014-03-04 00:42:33,654 [server.Accumulo] INFO : Connected to HDFS
Starting master on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting garbage collector on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting tracer on localhost
WARN : Max files open on localhost is 1024, recommend 65536
To confirm that it is up and running, view the Accumulo Overview page at: http://localhost:50095
I had ThriftSecurityException(user:root,
code:BAD_CREDENTIALS) error under Recent Logs and I fixed it by providing root
password in accumulo-site.xml trace.token.property.password. By default it is
set to “secret”. However, it is best to make a special user and password for
tracing and configure it.
To finish off, let’s create ACCUMULO_HOME like we did for
ZooKeeper, Java, etc.
$ sudo vim /etc/profile.d/accumulo.sh
export ACCUMULO_HOME=/usr/local/accumulo/
Tuesday, March 18, 2014
Installing Zookeeper
Think of ZooKeeper as coordination service for cluster environment managing distributed processes.
ZooKeeper ensures:
- Consistency
- Atomicity
- Reliability
- Timeliness
In order to install ZooKeeper, you need to download stable release from its official site, extract it and make sure that our Hadoop user can access it.
[root@localhost local]# cd /usr/local/
[root@localhost local]# wget http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
--2014-03-03 20:25:54-- http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
Resolving apache.osuosl.org... 140.211.166.134
Connecting to apache.osuosl.org|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17699306 (17M) [application/x-gzip]
Saving to: “zookeeper-3.4.6.tar.gz”
100%[======================>] 17,699,306 280K/s in 45s
2014-03-03 20:26:41 (387 KB/s) - “zookeeper-3.4.6.tar.gz” saved [17699306/17699306]
[root@localhost local]# tar xzf zookeeper-3.4.6.tar.gz
[root@localhost local]# mv zookeeper-3.4.6 zookeeper
[root@localhost local]# chown -R hduser:hadoop zookeeper
[root@localhost local]# ll
….
drwxr-xr-x. 10 hduser hadoop 4096 Feb 20 05:58 zookeeper
…..
[root@localhost local]# cat /usr/local/zookeeper/conf/zoo_sample.cfg >> /usr/local/zookeeper/conf/zoo.cfg
It should look like
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
Notice that ZooKeeper needs temporary directory to store its files
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
Let’s change dataDir to point to /app/zookeeper and make sure that we have this directory with the right access rights.
[root@localhost app]# mkdir -p /app/zookeeper
[root@localhost app]# chown hduser:hadoop /app/zookeeper/
Now let’s start ZooKeeper
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
And verify that it is indeed running
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
2014-03-03 20:53:21,869 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-03-03 20:53:21,881 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost.localdomain
2014-03-03 20:53:21,882 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper/bin/../zookeeper-3.4.6.jar:/usr/local/zookeeper/bin/../src/java/lib/*.jar:/usr/local/zookeeper/bin/../conf:
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.5.1.el6.x86_64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hduser
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/hduser
2014-03-03 20:53:21,887 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/tmp
2014-03-03 20:53:21,891 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1d672476
Welcome to ZooKeeper!
2014-03-03 20:53:22,255 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2014-03-03 20:53:22,293 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2014-03-03 20:53:22,448 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x1448aca9c6e0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Now let’s set up ZOOKEEPER_HOME
$ sudo vim /etc/profile.d/zookeeper.sh
export ZOOKEEPER_HOME= /usr/local/zookeeper/
Done!
ZooKeeper ensures:
- Consistency
- Atomicity
- Reliability
- Timeliness
In order to install ZooKeeper, you need to download stable release from its official site, extract it and make sure that our Hadoop user can access it.
[root@localhost local]# cd /usr/local/
[root@localhost local]# wget http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
--2014-03-03 20:25:54-- http://apache.osuosl.org/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
Resolving apache.osuosl.org... 140.211.166.134
Connecting to apache.osuosl.org|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17699306 (17M) [application/x-gzip]
Saving to: “zookeeper-3.4.6.tar.gz”
100%[======================>] 17,699,306 280K/s in 45s
2014-03-03 20:26:41 (387 KB/s) - “zookeeper-3.4.6.tar.gz” saved [17699306/17699306]
[root@localhost local]# tar xzf zookeeper-3.4.6.tar.gz
[root@localhost local]# mv zookeeper-3.4.6 zookeeper
[root@localhost local]# chown -R hduser:hadoop zookeeper
[root@localhost local]# ll
….
drwxr-xr-x. 10 hduser hadoop 4096 Feb 20 05:58 zookeeper
…..
Configuring ZooKeeper
Simply copy over sample config file to zoo.cfg[root@localhost local]# cat /usr/local/zookeeper/conf/zoo_sample.cfg >> /usr/local/zookeeper/conf/zoo.cfg
It should look like
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
Notice that ZooKeeper needs temporary directory to store its files
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
Let’s change dataDir to point to /app/zookeeper and make sure that we have this directory with the right access rights.
[root@localhost app]# mkdir -p /app/zookeeper
[root@localhost app]# chown hduser:hadoop /app/zookeeper/
Now let’s start ZooKeeper
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
And verify that it is indeed running
[hduser@localhost tmp]$ /usr/local/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
2014-03-03 20:53:21,869 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-03-03 20:53:21,881 [myid:] - INFO [main:Environment@100] - Client environment:host.name=localhost.localdomain
2014-03-03 20:53:21,882 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_51
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2014-03-03 20:53:21,885 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper/bin/../zookeeper-3.4.6.jar:/usr/local/zookeeper/bin/../src/java/lib/*.jar:/usr/local/zookeeper/bin/../conf:
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.5.1.el6.x86_64
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hduser
2014-03-03 20:53:21,886 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/hduser
2014-03-03 20:53:21,887 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/tmp
2014-03-03 20:53:21,891 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1d672476
Welcome to ZooKeeper!
2014-03-03 20:53:22,255 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2014-03-03 20:53:22,293 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost.localdomain/127.0.0.1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2014-03-03 20:53:22,448 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x1448aca9c6e0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Now let’s set up ZOOKEEPER_HOME
$ sudo vim /etc/profile.d/zookeeper.sh
export ZOOKEEPER_HOME= /usr/local/zookeeper/
Done!
Subscribe to:
Posts (Atom)