HADOOP SINGLE NODE INSTALLATION

    INSTALLATION GUIDE FOR HADOOP IN UBUNTU 12.04 (SINGLE NODE)

Installing Java-7-oracle

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java7-installer 

Read More

Yahoo! RAISE OF A LEGEND ?? PART 1

DECLINE OF A LEGEND 

Yahoo! is a technology powered media company which connects half a billion people around the world.It has always contributed a lot to the development of internet and has embraced open source technologies. It was once a legend in the web technology who lost its place.

Read More

INSTALLATION GUIDE FOR HADOOP IN UBUNTU 12.04 (SINGLE NODE)

    INSTALLATION GUIDE FOR HADOOP IN UBUNTU 12.04 (SINGLE NODE)

Installing Java-7-oracle

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java7-installer

Create a separate user for hadoop

$ sudo addgroup hadoop

$ sudo adduser --ingroup hadoop hduser

Configure SSH

su - hduser

ssh-keygen -t rsa -P ""

To be sure that SSH installation is went well, you can open a new terminal and try to create ssh session using hduser by the following command:

$ssh localhost

if localhost is not connected reinstalltion of ssh is needed

 

sudo apt-get install openssh-server

Edit Sudoers

pkexec visudo

Add below line to add hduser into sudoers

hduser (ALL)=(ALL) ALL

Ctrl + O to save nano 

and exit from the editor  

Disable IPv6

 

$sudo gedit /etc/sysctl.conf

This command will open sysctl.conf in text editor, you can copy the following lines at the end of the file:

#disable ipv6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

If you faced a problem telling you don't have permissions, just remember to run the previous commands by your root account.

These steps required you to reboot your system, but alternatively, you can run the following command to re-initialize the configurations again.

$sudo sysctl -p

To make sure that IPV6 is disabled, you can run the following command:

$cat /proc/sys/net/ipv6/conf/all/disable_ipv6

Configuration of Hadoop

Installing Hadoop

Now we can download Hadoop to begin installation. Go to Apache Downloads and download Hadoop version 1.0.4.(current stable version)

Then you need to extract the tar file and rename the extracted folder to 'hadoop'. Open a new terminal and run the following command:

$ cd /home/hduser

$ sudo tar xzf hadoop-0.20.2.tar.gz

$ sudo mv hadoop-0.20.2 hadoop

Update $HOME/.bashrc

You will need to update the .bachrc for hduser (and for every user you need to administer Hadoop). To open .bachrc file, you will need to open it as root:

$sudo gedit /home/hduser/.bashrc

Then you will add the following configurations at the end of .bachrc file

# Set Hadoop-related environment variables

export HADOOP_HOME=/home/hduser/hadoop

# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

# Some convenient aliases and functions for running Hadoop-related commands

unalias fs &> /dev/null

alias fs="hadoop fs"

unalias hls &> /dev/null

alias hls="fs -ls"

# If you have LZO compression enabled in your Hadoop cluster and

# compress job outputs with LZOP (not covered in this tutorial):

# Conveniently inspect an LZOP compressed file from the command

# line; run via:

#

# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo

#

# Requires installed 'lzop' command.

#

lzohead () {

    hadoop fs -cat $1 | lzop -dc | head -1000 | less

}

# Add Hadoop bin/ directory to PATH

export PATH=$PATH:$HADOOP_HOME/bin

hadoop-env.sh

We need only to update the JAVA_HOME variable in this file. Simply you will open this file using a text editor using the following command:

$sudo gedit /home/hduser/hadoop/conf/hadoop-env.sh

or

nano /home/hduser/hduser/hadoop/conf/hadoop-env.sh

Then you will need to change the following line

# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

To 

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

Note: if you faced "Error: JAVA_HOME is not set" Error while starting the services, then you seems that you forgot toe uncomment the previous line (just remove #).

core-site.xml

First, we need to create a temp directory for Hadoop framework. If you need this environment for testing or a quick prototype (e.g. develop simple hadoop programs for your personal test ...), I suggest to create this folder under /home/hduser/ directory, otherwise, you should create this folder in a shared place under shared folder (like /usr/local ...) but you may face some security issues. But to overcome the exceptions that may caused by security (like java.io.IOException), I have created the tmp folder under hduser space.

To create this folder, type the following command:

$ sudo mkdir  /home/hduser/tmp

Please note that if you want to make another admin user (e.g. hduser2 in hadoop group), you should grant him a read and write permission on this folder using the following commands:

$ sudo chown hduser:hadoop /home/hduser/tmp

$ sudo chmod 755 /home/hduser/tmp

Now, we can open hadoop/conf/core-site.xml to edit the hadoop.tmp.dir entry.

We can open the core-site.xml using text editor:

$sudo gedit /home/hduser/hadoop/conf/core-site.xml

or

nano /home/hduser/hduser/hadoop/conf/core-site.xml

Then add the following configurations between .. xml elements:

  hadoop.tmp.dir

  /home/hduser/tmp

  A base for other temporary directories.

  fs.default.name

  hdfs://localhost:54310

  The name of the default file system.  A URI whose

  scheme and authority determine the FileSystem implementation.  The

  uri's scheme determines the config property (fs.SCHEME.impl) naming

  the FileSystem implementation class.  The uri's authority is used to

  determine the host, port, etc. for a filesystem.

mapred-site.xml

We will open the hadoop/conf/mapred-site.xml using a text editor and add the following configuration values (like core-site.xml)

nano /home/hduser/hduser/hadoop/conf/mapred-site.xml

  mapred.job.tracker

  localhost:54311

  The host and port that the MapReduce job tracker runs

  at.  If "local", then jobs are run in-process as a single map

  and reduce task.

 

hdfs-site.xml

Open hadoop/conf/hdfs-site.xml using a text editor and add the following configurations:

nano /home/hduser/hduser/hadoop/conf/hdfs-site.xml

  dfs.replication

  1

  Default block replication.

  The actual number of replications can be specified when the file is created.

  The default is used if replication is not specified in create time.

 

Formatting NameNode

~/hduser/hadoop/bin/hadoop namenode -format

You should format the NameNode in your HDFS. You should not do this step when the system is running. It is usually done once at first time of your installation.

Run the following command

$/home/hduser/hadoop/bin/hadoop namenode -format

NameNode Formatting

Starting Hadoop Cluster

You will need to navigate to hadoop/bin directory and run ./start-all.sh script.

cd ~/hduser/hadoop/bin/

./start-all.sh

Starting Hadoop Services using ./start-all.sh

There is a nice tool called jps. You can use it to ensure that all the services are up.

Read More

ERRORS IN HADOOP INSTALLATION & HOW TO SOLVE THEM

TO ERR IS TO HUMAN 

And we cannot find anybody who haven't made an error while installing hadoop.Apache Hadoop is now a trade mark for open-source software for reliable, scalable and distributed computing.

Apache hadoop project includes one of the most important, Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.

Read More

DESKTOP CUSTOMIZATIONS IN WINDOWS

This post will help you turn your desktop into a productive and nice looking space which you can customize every day. The interactive panels provide specific information so you wont have anything unnecessary on your desktop.

TOOLS NEEDED:

• Rainmeter 1.2 r380 or higher
• Omnimo 2 for Rainmeter which can be downloaded from this link

Read More

WRITING HOPES WITH EYES

Its winter already the snow has began to fall  …Its again another season that passes  away in silence .

It was so silent  that I could not even hear my heart beat. Wait  is my heart beating? May or may not be?
 How do I know that?
 Did it turned out to be clay?
No I don’t think so

I think there is a war going in my brain between my neurons ….

May be my immunity system is working   so hard to preserve me like this...

No.

Read More

DREAM BIG.....

DREAM ..DREAM..DREAM... DARE  TO  DREAM...


It was in 2007 that the film OM SANTHI OM  was released .....
if you think  whats the connection between this movie and the DREAMS..its just simple...

"One life time is not enough for all the dreams to come true.." and "Agar kisi cheez ko dil se chaaho to puri kayanat usey tumse milane ki koshish mein lag jaati hai."

Read More

The girl who feared happiness - Shot -3

 Lullaby - a song unsung

Its spring again but I can only wonder what my last spring was like.

My disease have grown so strong that I don't know where I am heading.. I can now remember only those things which leaves a trail. I am still that girl who has grown so strong against happiness.

Life is still giving me twists and turns..

Its like expecting the unexpected..

When you take a step, your life is ready to give surprises .and you keep on guessing what twist it is going to bring in. This guessing game has gone too far that I don't find any interest in this game anymore.. why don't my life give me some exciting puzzles. the one which can make my heart thump up with joy..

Now that i know what my life is intending to do with me .. I am enjoying it to the fullest... In between I have started to enjoy all the small things, stealing away from the dark shadows of my life ..

I feel happy when i can predict whats the next twist that my life is planning to give me ..

One thing I learned is take what ever you can from this life ..

Live the way you want it to be..

Express your views & thoughts, may be others will be able to understand you.

Even though it may not be appropriate, its ok to confess.

May be life has a way of giving back everything it has taken from you.  Even the saddest and the happiest memories will be recreated.. I have noticed one more thing about my life and that is my inability to remember those incidents which could have been bad for me .. Never once my life took away my memories of happiness .. It gives me a big relief that my life is so considerate of me ..

It never fails to astonish me !!

Always putting things that are needed and valuable in life and removing those junks that could pollute my thoughts. Even though I cry remembering the path that life makes me walk .. I can assure myself it was much needed  as I was able to see only the goodness around me ..

Even when I curse my life of becoming allergic to happiness it encouraged me to see the world through a different light.. Happiness is not something that can be captured.. The more you try to run behind it , it eludes.

I am happy that I can enjoy each and every moments in life with out thinking whats going to happen next; even though  the  feeling of badness still lingers but i have confidence that once i sleep and wake up I will forget it ..

Each night I wait for the life to sing its lullaby ..

I am enjoying the song .. 

"Sleep and I forget everything .."

Read More

The girl who feared happiness - Shot -2

 Exploring the unknown.

Life was  a mystery and I was trying to solve the puzzle.. 

There he was coming in as late as usual to attend our annual higher secondary creative program meeting... He sat opposite to me and spread out a warm smile. His smile was enough for me not to get close to him . I tend not to notice him and involved in the discussion .. But sooner or later destiny had to put us together in various awkward situations ... It always made me think how he was able to make me smile and remove the slightest fear or agony.. I was back to being normal ...

I just wished upon a star not to break my dream ...

Let me live with those sweet memories ... Least I want them to be tarnished with the monsters of my mind...

Two people from different backgrounds with different personalities came together.. Shared their worlds, fears & happiness. The solace the two hearts found was astonishing.

The rapport, the banter and stealing glances.. those made her heart beat..

Yet never uttered the word love to each other.

Both were afraid. He was afraid if this will turn out to be like his previous affair  and she was afraid if it hurts falling in love. 

Everyday I waved him goodbye standing in my balcony. Until I had to change my hostel. That was a private moment which I cherished. We talked, argued, fought and still I found solace in him.  I could talk anything.. any stupid thoughts.

He would answer my queries like answering a little child.

He told her many times he like her.. But when she tried to confirm it , he always turned it as a joke.  He was afraid if she will walk away from him

She wanted assurance and he thought she may reject him.

It  was really a tormenting feeling .

The heart knows what that feeling was but my brain and my memories refused to believe them.

We both lacked the courage to pursue the desires of heart and left it unresolved to the very end.

My unusual life which always brought up some troubles and twists in my life .. I prayed one last time not to torment me any more and bring a twist which can make my unusual boring life to normal .... with him.

But I was asking for more than what I deserved. 

But life had another twist to play and we had to part ways.. He had to go back to his native and I was left alone.

Bidding a good bye was heart wrenching. I no longer had the strength to endure it.

On his last day at our school.. I intended to visit him and bid a good bye.  When I asked him if it will create any scene ..

I was not pointing at the gossips or other problems, instead I was wondering if I will be creating a scene crying my heart out. I don't know if I will be able to bid him a farewell with out crying my heart out. I thought it is better to leave him as such.. It was better if we don't meet again. I decided not to meet him one last time 

That was how I planned to end our journey .. but to my dismay my life had other plans and I had to go to his place and seeing him stand there I had no choice but to  bid a quick bye .. And I clearly remember my cracked voice .. I was finding it difficult to talk .. breath was short and I had to get away from him .. Never will he know that I will forget everything about him once I sleep and wake up. I will have to force myself to rack my brains and  remember what was inflicting me with this pain.  I am thanking my self for being inflicted with this pain ..but at the same time I fear if I will forget him just like I forget everything 

I was very sure I wont be seeing him again in my life. Even when he asked me if everything was alright I was crying my heart out silently weeping so that my roommate doesn't hear me .. 

As always said in my matter.. the  fun filled journey had to put an abrupt stop as my demons took away my mind... I tried hard not to give in .. But there was no  stopping them ... And I was back to being the allergic girl again ... I was contemplated on how my life was nothing but a sanctuary for the darkness..

I learned they he was an anchor which I can't use anymore to keep my demons at bay..

They had grown stronger ..And abruptly I ended it there.. I don't want to forget anything but my heart was reckoning me to thank him for making me feel like I am normal just like others.. Thank you..

Read More

The girl who feared happiness - Shot - 1

Sleep and I forget everything ..

It has been like this for years.. I don' know exactly when I was infected with this strange disease.

But each day when I wake up, its like a brand new chapter.. I just forget those events from yesterdays which should have been crystal clear .

I really don't seem to remember what I did but people always sound funny.. They accuse me of something which I could not recall. There was that weird feeling of being accused that made me to look into myself and understand the abnormalities in me.
Those incidents continued and my tears on  my pillow spoke the words for me. I understood the emptiness in my life. My life was blocking out my memory of yesterdays ..

The funniest part - "Not everything".

I could not understand what it was like to be cheated by the memories.

I wanted to scream..

beg for my memories ..

Please leave my memories with me .. I wanted to look into them, understand my mistakes, rectify them and move on.

Give me a chance to change myself.

But my memories were locked away. 

How can it erase all those memories when I stood there like a laughing stock.

Uncertainties in life - I hate those..

Not knowing the emotional stress that you encountered the previous day, washed away with the tides of tears. It was becoming tedious.

When this continued, I had no option but to wait and watch what was in store for me. Some days I cried and slept like a baby, tried of everything that  my life was pounding me with. It was always like that; when everything was going smooth, my life would throw a fit. With out a twist my journey would be incomplete.

Moments like those made me fear about how my journey will be, how it will be completed with the twists and turns.

I was afraid, if I woke up from the dreams..

It  will be destroyed and once again I will be coerced to enter the fearful dreaded reality ..

I was far well off from the reality and loved to live in the dreams but dreams were not always the solace. I wanted to be like others and wanted to embrace the warmth and solace of happiness. But it was far from me.

In between all those emotional dramas, I was very happy when I got my little kitty.. All day I would play with it and its company made my heart melt. By then I was aware of my life's mysterious way of holding me back from my happiness.

It was on the eve of  a new year, my little kitty was called back and I kept crying knowing that I am to forget the impact it had on my little heart. I was going to forget all the moments I had with my catty and I wont remember anything.

Slowly but steadily my happiness were taken away by those uncertainties  and the hurdles in my life. 

Instead of striking back, my soul was crushed and I was becoming allergic to .... happiness.

Read More

Oru Yatra

Read More