Hadoop Installation using a single Node cluster

Overview:

Hadoop is a technology which enables distributed processing of large set of data sets across clusters ranging from 1 server to thousands of server ensuring a high degree of Fault Tolerence.

Hadoop is a framework which consists of following basic modules

1) Hadoop Common – contains libraries and utilities needed by other Hadoop modules
2) Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on the commodity machines, providing very high aggregate bandwidth across the cluster.
3) Hadoop YARN – a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users’ applications.
4) Hadoop MapReduce – a programming model for large scale data processing

In this Tutorial we will install & configure Hadoop on Ubuntu(12.10/13.04/13.10).Follow the below steps :

Step:1 Update your machine

root@hadoop1:~# apt-get update

Install python-software-properties module

root@hadoop1:~# apt-get install python-software-properties

Add the sun – java repository

root@hadoop1:~# add-apt-repository ppa:webupd8team/java 
root@hadoop1:~# apt-get update && sudo apt-get upgrade
root@hadoop1:~# apt-get install oracle-java6-installer

Check the installed java version

root@hadoop1:~# java -version 
java version "1.6.0_45" 
Java(TM) SE Runtime Environment (build 1.6.0_45-b06) 
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

If there are 2 versions of java as seen from the update-java-alternatives command

Then run the following command to set this java to the latest versions

root@hadoop1:~# update-alternatives --config java

There is only one alternative in link group java: /usr/lib/jvm/java-6-oracle/jre/bin/java
Nothing to configure.

Step:2  Adding a group to the system

root@hadoop1:~#  addgroup hadoopgroup

Adding a hadoop user to the earlier created group

root@hadoop1:~# adduser --ingroup hadoopgroup hadoopuser 
Adding user `hadoopuser' ... 
Adding new user `hadoopuser' (1001) with group `hadoopgroup' ... 
Creating home directory `/home/hadoopuser' ... 
Copying files from `/etc/skel' ... 
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully 
Changing the user information for hadoopuser 
Enter the new value, or press ENTER for the default 
Full Name []: HADOOP USER 
Room Number []: 
Work Phone []: 
Home Phone []: 
Other []: 
Is the information correct? [Y/n] Y

 Step:3 Create Passwordless authentication

root@hadoop1:~# su - hadoopuser
hadoopuser@hadoop1:~$ ssh-keygen -t rsa -P "" 
Generating public/private rsa key pair. 
Enter file in which to save the key (/home/hadoopuser/.ssh/id_rsa): 
Created directory '/home/hadoopuser/.ssh'. 
Your identification has been saved in /home/hadoopuser/.ssh/id_rsa. 
Your public key has been saved in /home/hadoopuser/.ssh/id_rsa.pub. 
The key fingerprint is: 
82:a0:cb:f4:fa:1f:ac:f5:29:54:34:e7:56:ee:b0:9f hadoopuser@hadoop1.example.com 
The key's randomart image is: 
+--[ RSA 2048]----+ 
|                 | 
|       o . .     | 
|  .   . + o      | 
| . . . . + .     | 
|..  . o S +      | 
|o.. .. . . .     | 
|.. ..+    . .    | 
|  . o.o .  E     | 
| ..o...o         | 
+-----------------+ 
hadoopuser@hadoop1:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys 
hadoopuser@hadoop1:~$ ssh localhost 
we will get below message after login
The authenticity of host 'localhost (127.0.0.1)' can't be established. 
ECDSA key fingerprint is ee:be:18:ef:e6:3d:e3:8d:8a:17:ca:d1:a3:d6:d6:49. 
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. 
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64)

Step:4 Disable ipv6

As a root Append the file /etc/sysctl.conf and add the following lines

root@hadoop1:~# vi /etc/sysctl.conf
# disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1 
Save & exit
root@hadoop1:~# sysctl -p 
net.ipv6.conf.all.disable_ipv6 = 1 
net.ipv6.conf.default.disable_ipv6 = 1 
net.ipv6.conf.lo.disable_ipv6 = 1

To verify the ipv6 has been disabled

root@hadoop1:~# cat /proc/sys/net/ipv6/conf/all/disable_ipv6 
1

Step:5 Adding the hadoop repository

root@hadoop1:~# add-apt-repository ppa:hadoop-ubuntu/stable

Update and upgrade

root@hadoop1:~# apt-get update && apt-get upgrade

Step:6 Now Install Hadoop

root@hadoop1:~# apt-get install hadoop

Verify the hadoopuser details

root@hadoop1:~# id hadoopuser 
uid=1001(hadoopuser) gid=1002(hadoopgroup) groups=1002(hadoopgroup)

Add the hadoop user to the sudo file so that it will have the root level permissions

root@hadoop1:~# visudo
and add the following line :
hadoopuser ALL=(ALL:ALL) ALL

Set the environment in the hadoop user .bashrc file as follows

root@hadoop1:~# vi /home/hadoopuser/.bashrc 
and add the following line
# Set Hadoop-related environment variables   
export HADOOP_HOME=/home/hadoopuser/hadoop  
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)`
export JAVA_HOME=/usr/lib/jvm/java-6-oracle/ 
# Add Hadoop bin/ directory to PATH  
export PATH=$PATH:$HADOOP_HOME/bin  
export PATH=$PATH:/usr/lib/hadoop/bin/

#Set some aliased

unalias fs &> /dev/null   
alias fs="hadoop fs"    
unalias hls &> /dev/null  
alias hls="fs -ls"

Step:7 Now Configure the Hadoop

root@hadoop1:~# chown -R hadoopuser:hadoopgroup /var/log/hadoop/ 
root@hadoop1:~# chmod -R 755 /var/log/hadoop/ 
root@hadoop1:~# cd /usr/lib/hadoop/conf/ 
root@hadoop1:/usr/lib/hadoop/conf# ls -ltr 
total 76 
-rw-r--r-- 1 root hadoop  382 Mar 24  2012 taskcontroller.cfg 
-rw-r--r-- 1 root hadoop 1195 Mar 24  2012 ssl-server.xml.example 
-rw-r--r-- 1 root hadoop 1243 Mar 24  2012 ssl-client.xml.example 
-rw-r--r-- 1 root hadoop   10 Mar 24  2012 slaves 
-rw-r--r-- 1 root hadoop   10 Mar 24  2012 masters 
-rw-r--r-- 1 root hadoop  178 Mar 24  2012 mapred-site.xml 
-rw-r--r-- 1 root hadoop 2033 Mar 24  2012 mapred-queue-acls.xml 
-rw-r--r-- 1 root hadoop 4441 Mar 24  2012 log4j.properties 
-rw-r--r-- 1 root hadoop  178 Mar 24  2012 hdfs-site.xml 
-rw-r--r-- 1 root hadoop 4644 Mar 24  2012 hadoop-policy.xml 
-rw-r--r-- 1 root hadoop 1488 Mar 24  2012 hadoop-metrics2.properties 
-rw-r--r-- 1 root hadoop 2237 Mar 24  2012 hadoop-env.sh 
-rw-r--r-- 1 root hadoop  327 Mar 24  2012 fair-scheduler.xml 
-rw-r--r-- 1 root hadoop  178 Mar 24  2012 core-site.xml 
-rw-r--r-- 1 root hadoop  535 Mar 24  2012 configuration.xsl 
-rw-r--r-- 1 root hadoop 7457 Mar 24  2012 capacity-scheduler.xml 
root@hadoop1:/usr/lib/hadoop/conf#

But before we start using them, we need to modify several files in the /conf folder.

hadoop-env.sh

replace the JAVA_HOME line with the below line.
export JAVA_HOME=/usr/lib/jvm/java-7-oracle

core-site.xml 
Create a base for temporary files
root@hadoop1:/usr/lib/hadoop/conf# mkdir /home/hadoopuser/tmp 
root@hadoop1:/usr/lib/hadoop/conf# chown hadoopuser:hadoopgroup /home/hadoopuser/tmp/ 
root@hadoop1:/usr/lib/hadoop/conf# chmod 755 /home/hadoopuser/tmp/ 
root@hadoop1:/usr/lib/hadoop/conf#

Replace the original contents with

root@hadoop1:/usr/lib/hadoop/conf# cat core-site.xml 
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<!-- Put site-specific property overrides in this file. --> 
<configuration> 
<property> 
<name>hadoop.tmp.dir</name> 
<value>/home/hadoopuser/tmp</value> 
<description>A base for other temporary directories.</description> 
</property> 
<property> 
<name>fs.default.name</name> 
<value>hdfs://localhost:54310</value> 
<description>The name of the default file system.  A URI whose  scheme and authority determine the FileSystem implementation.  The  uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class.  The uri's authority is used to 
determine the host, port, etc. for a filesystem.</description> 
</property> 
</configuration>

 

root@hadoop1:/usr/lib/hadoop/conf# cat mapred-site.xml
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<!-- Put site-specific property overrides in this file. --> 
<configuration> 
<property> 
<name>mapred.job.tracker</name> 
<value>localhost:54311</value> 
<description>The host and port that the MapReduce job tracker runs 
at.  If "local", then jobs are run in-process as a single map 
and reduce task. 
</description> 
</property> 
</configuration>

 

root@hadoop1:/usr/lib/hadoop/conf# cat hdfs-site.xml 
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<!-- Put site-specific property overrides in this file. --> 
<configuration> 
<property> 
<name>dfs.replication</name> 
<value>1</value> 
<description>Default block replication. 
The actual number of replications can be specified when the file is created. 
The default is used if replication is not specified in create time. 
</description> 
</property> 
</configuration>

root@hadoop1:/usr/lib# chown -R hadoopuser:hadoopgroup /usr/lib/hadoop/ 
root@hadoop1:/usr/lib# ls -ltr hadoop/ 
total 16 
lrwxrwxrwx 1 hadoopuser hadoopgroup   15 Apr 24  2012 pids -> /var/run/hadoop 
lrwxrwxrwx 1 hadoopuser hadoopgroup   15 Apr 24  2012 logs -> /var/log/hadoop 
lrwxrwxrwx 1 hadoopuser hadoopgroup   41 Apr 24  2012 hadoop-tools-1.0.2.jar -> ../../share/hadoop/hadoop-tools-1.0.2.jar 
lrwxrwxrwx 1 hadoopuser hadoopgroup   40 Apr 24  2012 hadoop-test-1.0.2.jar -> ../../share/hadoop/hadoop-test-1.0.2.jar 
lrwxrwxrwx 1 hadoopuser hadoopgroup   44 Apr 24  2012 hadoop-examples-1.0.2.jar -> ../../share/hadoop/hadoop-examples-1.0.2.jar 
lrwxrwxrwx 1 hadoopuser hadoopgroup   40 Apr 24  2012 hadoop-core.jar -> ../../share/hadoop/hadoop-core-1.0.2.jar 
lrwxrwxrwx 1 hadoopuser hadoopgroup   40 Apr 24  2012 hadoop-core-1.0.2.jar -> ../../share/hadoop/hadoop-core-1.0.2.jar 
lrwxrwxrwx 1 hadoopuser hadoopgroup   39 Apr 24  2012 hadoop-ant-1.0.2.jar -> ../../share/hadoop/hadoop-ant-1.0.2.jar 
lrwxrwxrwx 1 hadoopuser hadoopgroup   26 Apr 24  2012 contrib -> ../../share/hadoop/contrib 
lrwxrwxrwx 1 hadoopuser hadoopgroup   16 Apr 24  2012 conf -> /etc/hadoop/conf 
drwxr-xr-x 9 hadoopuser hadoopgroup 4096 Dec 15 05:16 webapps 
drwxr-xr-x 2 hadoopuser hadoopgroup 4096 Dec 15 05:16 libexec 
drwxr-xr-x 2 hadoopuser hadoopgroup 4096 Dec 15 05:16 bin 
drwxr-xr-x 3 hadoopuser hadoopgroup 4096 Dec 15 05:16 lib 
root@hadoop1:/usr/lib# 
root@hadoop1:/etc/hadoop/conf# chown -R hadoopuser:hadoopgroup /etc/hadoop/ 
root@hadoop1:/etc/hadoop/conf# 
root@hadoop1:~# su - hadoopuser 
hadoopuser@hadoop1:~$ mkdir hadoop

Commands To Manage Hadoop Services:

  • start-dfs.sh – Starts the Hadoop DFS daemons, the namenode and datanodes. Use this before start-mapred.sh
  • stop-dfs.sh – Stops the Hadoop DFS daemons.
  • start-mapred.sh – Starts the Hadoop Map/Reduce daemons, the jobtracker and tasktrackers.
  • stop-mapred.sh – Stops the Hadoop Map/Reduce daemons.
  • start-all.sh – Starts all Hadoop daemons, the namenode, datanodes, the jobtracker and tasktrackers. Deprecated; use start-dfs.sh then start-mapred.sh
  • stop-all.sh – Stops all Hadoop daemons. Deprecated; use stop-mapred.sh then stop-dfs.sh

Format the hadoop file system

root@hadoop1:/usr/lib/hadoop/conf# su - hadoopuser 
hadoopuser@hadoop1:~$ hadoop namenode -format 
13/12/15 19:53:16 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************ 
STARTUP_MSG: Starting NameNode 
STARTUP_MSG:   host = java.net.UnknownHostException: hadoop1.example.com: hadoop1.example.com 
STARTUP_MSG:   args = [-format] 
STARTUP_MSG:   version = 1.0.2 
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012 
************************************************************/ 
13/12/15 19:53:17 INFO util.GSet: VM type       = 64-bit 
13/12/15 19:53:17 INFO util.GSet: 2% max memory = 19.33375 MB 
13/12/15 19:53:17 INFO util.GSet: capacity      = 2^21 = 2097152 entries 
13/12/15 19:53:17 INFO util.GSet: recommended=2097152, actual=2097152 
13/12/15 19:53:37 INFO namenode.FSNamesystem: fsOwner=hadoopuser 
13/12/15 19:53:37 INFO namenode.FSNamesystem: supergroup=supergroup 
13/12/15 19:53:37 INFO namenode.FSNamesystem: isPermissionEnabled=true 
13/12/15 19:53:37 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 
13/12/15 19:53:37 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 
13/12/15 19:53:37 INFO namenode.NameNode: Caching file names occuring more than 10 times 
13/12/15 19:53:58 INFO common.Storage: Image file of size 116 saved in 0 seconds. 
13/12/15 19:53:58 INFO common.Storage: Storage directory /home/hadoopuser/tmp/dfs/name has been successfully formatted. 
13/12/15 19:53:58 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************ 
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: hadoop1.example.com: hadoop1.example.com 
************************************************************/

Starting the Hadoop Cluster using start-all.sh 

hadoopuser@hadoop1:~$ start-all.sh 
Warning: $HADOOP_HOME is deprecated. 
starting namenode, logging to /usr/lib/hadoop/libexec/../logs/hadoop-hadoopuser-namenode-hadoop1.example.com.out 
localhost: starting datanode, logging to /usr/lib/hadoop/libexec/../logs/hadoop-hadoopuser-datanode-hadoop1.example.com.out 
localhost: starting secondarynamenode, logging to /usr/lib/hadoop/libexec/../logs/hadoop-hadoopuser-secondarynamenode-hadoop1.example.com.out 
starting jobtracker, logging to /usr/lib/hadoop/libexec/../logs/hadoop-hadoopuser-jobtracker-hadoop1.example.com.out 
localhost: starting tasktracker, logging to /usr/lib/hadoop/libexec/../logs/hadoop-hadoopuser-tasktracker-hadoop1.example.com.out 
hadoopuser@hadoop1:~$

or run

start-dfs.sh  
start-mapred.sh

To check if hadoop is running or not, use the below command

hadoopuser@hadoop1:~$ jps 
35401 NameNode 
35710 JobTracker 
35627 SecondaryNameNode 
35928 Jps 
hadoopuser@hadoop1:~$

Leave a Reply

Your email address will not be published. Required fields are marked *