Basic unix commands

Bellow are some basic linux commands required for setting up the hadoop environment.

$hostname
This command will display the name of your host

$sudo su
This command is used for login as super user

$addgroup  name of group
This command is used for adding a group

$adduser  name of user
This command is used for adding a user

$ls
This command is used for display the list of files inside a directory

$pwd
This command is used for display the present working directory

$cd
This command is used for changing the directory

$mkdir
This command is used for creating a  directory

$touch filename
This command is used for creating a blank file

$kill -9 processId

This command is used for manually killing a running process

$who 
This command is used for knowing that who is logged in the system

$df
This command is used to to how much disk space is available

Unix Configuration Before Hadoop instalation

In this post i am going to teach you how to install hadoop in your windows system

Prerequisite to Hadoop Installation

You have installed Ubuntu 16.04 Desktop version in your Virtual Machine 12
You have installed Java(jdk 1.8) in your Ubuntu system.
JAVA_HOME=/usr/local/java/jdk1.8.0_91
Check your hostname is Ubuntu, run the bellow command to check your host name
    $hostname  #output should be Ubuntu
    you need to configure your linux before installing hadoop
 
    follow the bellow commands
    Login as root user
 
    $sudo su
    $whoami   #output should be root

    Add a group and a user respectively  by using bellow command,same group and user names will be  used through out complete hadoop posts.
    #sudo addgroup hadoopdevlopergroup
    # sudo adduser hadoopdevloperuser

  while executing adduser command it will ask for password 2 times ,provide a password and proceed
    add hadoopdevloperuser to  hadoopdevlopergroup
    #sudo adduser hadoopdevloperuser  hadoopdevlopergroup
 
    Now hadoopdevloperuser   need to be added as super user so that it can have super user access
    #sudo vi sudo
    Add the bellow line
    hadoopdevloperuser  ALL=(ALL)ALL
    Save the file and exit,shut down the machine and login as newly added user
    

    Configuring SSH

hadoop requires ssh access to access its nodes means remote machine and local machine,install ssh server on machine by using bellow command,open terminal (ctrl+t is shortcut for opening terminal)

    $sudo apt-get install open-ssh server
    enter the password and then Y to continue
    Generate SSH for communication:use bellow command for generating ssh
    $ssh-keygen
    press enter
    Now copy the public key to the authorized_keys file, so that ssh should not require passwords every time
    $cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    Change permissions of the authorized_keys file to have all permissions for hadoopdevloperuser
    Start your SSh
    $ sudo /etc/init.d/ssh  restart
    Test the SSH connectivity
    $ ssh  localhost
   Type YES if it ask for then you will be able to connect without asking any password
   You need to disable IPV6 in order to use hadoop
   $ sudo vim /etc/sysctl.conf
   Enter the password and add bellow lines in file and save
  # disable ipv6   net.ipv6.conf.all.disable_ipv6 = 1   net.ipv6.conf.default.disable_ipv6 = 1   net.ipv6.conf.lo.disable_ipv6 = 1

Check if IPv6 is disabled.
After a system reboot the output of
hduser@ubuntu:~$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
should be 1, meaning that IPV6 is actually disabled. Without reboot it would be showing you 0.