How to solve Datanode error in Hadoop

Sometimes the datanode daemon process does not start successfully when you run the startall.sh script in hadoop on both the single and multinode clusters.

The Datanode log gives java.io.IOException .Further it explains that namespaceid of namenode and datanode are not compatible.

This problem can be averted by changing the namesapce id of the datanode.

To change the namespace id of datanode follow these steps

1. Check the value of hadoop.tmp.dir in conf/core-site.xml file. 
2. As my value is /app/hadoop/tmp i will open that directory.
3. Now /app/hadoop/tmp/dfs/name/current/VERSION represents the version file of name node which contains its namenode.
4. /app/hadoop/tmp/dfs/data/current/VERSION contains the namespace id of datanode.
5. Both of these namespace ids need to match.
6. Delete the file /app/hadoop/tmp/dfs/data/current/VERSION and restart the namenode.
7. Now Datanode will also start normally.

The  Version file looks something like this

    

Configure eclipse map reduce plugin for hadoop

Hadoop Eclipse Plug-in can be downloaded from this link .

The Hadoop Eclipse Plug-in provides tools to ease the experience of Map/Reduce on Hadoop. Among other things, the plug-in provides support to:

1. create Mapper, Reducer, Driver classes;
2. browse and interact with distributed file systems;
3. submit jobs and monitor their execution.

Now to configure the plugin you need to follow these steps

1. Move  the jar file you have downloaded in the plugins folder of your eclipse installation.
2. Restart the Eclipse and switch to  map-reduce perspective from Window-> open perspective ->other 
3. Now set your DFS location by selecting create new location in Map/Reduce locations area.
4. Enter the name of your location and specify the ports.
5. Add the port numbers of JobTracker and DFS as specified in conf/mapred-site.xml and  conf/core-site.xml files.

  6.  Now create new map-reduce project and configure your Hadoop installation directory. 

  7. Now you can start off by creating mapper and reducer classes.

  8. To run your project select run on Hadoop in the run configuration.

Interview Transcript

Background of the interview:

HCL techno-HR Interview at MSIT on 26 September 2012

I was the last person of 174 to be interviewed by HCL. I waited for more than 7 hours before I was called in the waiting room. The interviewer was grilling another candidate at that time. So, I was waiting outside sitting on a chair cracking veru bad jokes with others. After 5-10 minutes the ongoing interview ended and I asked my coordinator whether I should go in? The interviewer had just started talking on his phone and my wait went on for another 15 minutes before someone intervened him and I was asked to go in.

Me: May I come in sir?

Interviewer: come in.

Me: Good Evening Sir. May I sit?

Interviewer: baithiye sir (yes he did call me sir)

I sat down and gave him my resume. After seeing my name he asked me from where was I?

Me: Sir my grandparents are from Rajasthan while I was brought up in Delhi.

Interviewer: introduce yourself.

Me: Sir I think of myself as a creative guy. I try to search for ideas in real life situations so that I can implement them in a project. (I gave him an example of an android app) I recently created an android app for my mother. She is religious and uses rosary beads regularly. She often carries them with herself. I asked her why do you need to carry them when you have a smart phone so I created an app called Malaram (explained on my resume) for her.(short pause)This app has more than 500 downloads on market.

Interviewer: ok. Which of these technologies .NET, java or oracle are you comfortable with?

Me:   I am most comfortable in java.

Interviewer:  Do you know about databases?

Me: yes sir I have the basic understanding about the databases.

Interviewer: Do you know about web frameworks?

Me: yes sir.

Interviewer: Can you tell me how Google works?

Me: Sir Google works on map reduce algorithm. It distributes its searches in various nodes to deliver quick results in real time.

Then I explained a bit about map-reduce algorithm to him.

He did not expect such an answer. He was expecting something in a different domain.

Interviewer: tell me what will happen if I search my website on Google

Me: I started repeating map reduce algorithm with changed input string.

Interviewer: He stopped me and asked what is seo?

Me: search engine optimization.

Interviewer: how do you do that?

Me: we register our websites on search engines like google and bing

Interviewer: have you heard of bots?

Me: what sir?

Interviewer: yahoo bots, Google bots

Me: no sir.

Interviewer: hmm do you know about web-development?

Me: yes sir.

Interviewer: what is "br" used for?

Another phone call came for him in the middle of question and I waited again for a minute.

Me: sir break is used to introduce a line-break in the webpage

Interviewer: what is "tr"?

Me: sir it is used for table rows.

Interviewer: what is CSS?

Me: compact style sheets.......no sir cascaded style sheets.

Interviewer: what is its use?

Me: sir it is used for styling our webpages.

Interviewer: why is it not coded in html file?

Me: it keeps the code clean.

Interviewer: what about reuse-ability?

Me: yes sir that is also a use of CSS files.

Interviewer: He saw a strange url (goo.gl/d9Xdz  )on my resume and asked about it?

Me: Sir it is a shortened url for my web project.

After seeing my resume he asked me about one of the conferences attended by me

Interviewer: where was this Harvard us-India initiative held?

Me: it was held in Noida but organized by Harvard students

Interviewer: how did you manage to go there?

Me: sir I stumbled upon it while surfing and filled the form which consisted of some essays and got selected.

Interviewer: tell me a SQL query-> how will you print employees with dob less than a certain date?

Me: I gave him the right answer.
Then he said thank you and asked me to sign the sheet and go.
Me: thanks sir.

After 2 hours result was announced and 70 people were selected including me.

Working with Hadoop under Eclipse

Last week i was setting up development environment for Hadoop under the Eclipse IDE by following the instructions provided by Apache  . But the installation wasn't as smooth as one would hope. Some hassles were there which are not documented (or maybe were invisible to me). Anyways i am writing about the errors faced by me in this blog.

Maven installation

• you need maven to run the mvn command.
• While the documentation at maven download page is given for Windows 2000/XP or UNIX based systems.But it does not map correctly on win 7.
• In Windows 7 one need to make the  M2_HOME and M2 environment variables (see the installation instructions here) as system variables and then they should be added to the path.
• Other instructions are same as documented in release notes by APACHE.

 Now after running the $ mvn install -DskipTests command the building of the projects                started. But one of the projects Apache Hadoop Common Project failed to build and reported MojoFailureException.

• You need to install protobuf (protocol buffers) and add it to your path variable to build the project successfully.Download it from here.

Now the next project to fail in build was hadoop-yarn-common.

• The build process failed because windows was not able to run "saveVersion.sh" file successfully.
• You need to add following patch to the Pom.xml file in hadoop yarn common to allow windows to run the  "saveVersion.sh" file successfully.

After these steps the build process will complete itself successfully.and you can run the next command to create eclipse project.

$ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true

Now the projects will be ready to be imported in eclipse.

Hope these points would save other developers some time in installation.:-)

FBcoverprofilecombo

This post explains the Html5 and JavaScript app Create Facebook cover and profile pic combo.The major elements used in this app are Canvas and Drag and drop API of Html5.

Drag and Drop API is used to upload the images from the user.It makes the look and feel of the app trendy and reliable.Following event handler in JavaScript does that dirty work:

 dropzone.on('drop', function(e) {  
           //prevent browser from open the file when drop off  
           e.stopPropagation();  
           e.preventDefault();  
           dropzone.removeClass('hover');  
           //retrieve uploaded files data  
           var files = e.originalEvent.dataTransfer.files;  
           processFiles(files);  
           return false;  
      });  

Now the file is processed to create a cover image with the use of Canvas element of HTML5 in following function.The image settings are set depending on the field chosen (page or profile).Canvas element does the task of resizing the image.

 var getCanvasImage = function(image) {  
           //define canvas  
           var croping = $('input[name=croping]:checked').val();  
         //check for page or profile  
           if(croping == 'crop') {  
                imgWidth = 958;             imgHeight = 450;  
           }else{  
           imgWidth = 958;            imgHeight = 419;  
           }  
           var canvas = document.createElement('canvas');  
           canvas.width = imgWidth;  
           canvas.height = imgHeight;  
           var ctx = canvas.getContext('2d');  
           //draw canvas image       
           ctx.drawImage(image,0,0,image.width,image.height,0,0,imgWidth,imgHeight);   
           //convert canvas to jpeg url  
           return canvas.toDataURL("image/jpeg");  
      }  

After the image is created The above function returns the source of the cover image.Now this image is processed to create profile photo in the following event handler and later appended to the body of document.

 createprofile.on('click',function(){  
           var cw,ch,a,sx,sy;  
      var croping = $('input[name=croping]:checked').val();  
      if(croping == 'crop') {  
                cw=180;ch=180;sx=28;sy=240;  
           }else{  
           cw=190;ch=190;sx=28;sy=220;  
           }  
           var covershhh=document.getElementById("cov");  
           var canvs = document.createElement('Canvas');  
           canvs.width = cw;  
           canvs.height = ch;  
           var ctx = canvs.getContext('2d');  
           //draw canvas image                           
           ctx.drawImage(covershhh, 28, sy, cw, ch, 0, 0, cw, ch);
           var oImg=document.createElement('img');       
           oImg.id="gdfg";                      
           var a=canvs.toDataURL("image/jpeg");  
           oImg.src=a;
           document.getElementById("result2").appendChild(oImg);           
      });  

Note:All the measurements are taken are in pixels.In next post i will share some of the creative timelines created by this app.Till then good bye!!

Don't forget to give your feedback and encouragement.

Create facebook cover and profile pic combo

This post explains the Html5 and JavaScript app "Create Facebook cover and profile pic combo" developed by me this summer.Since the conception of timeline feature in Facebook most people  were initially very intrigued by it but later they were despised by it.Then i saw some creative examples of combining cover and profile pictures which aroused my interest in the timeline again.

So in order to join the cool gang i tried my hand at Photoshop with some disastrous results.Finally i decided to automate the whole process and hence after some smart coding and rigorous testing i created this app.

The best way to make most of this app is to keep your face in the lower left corner of the picture you want to upload on the web-app.This way your face will be visible in your profile pic.

You just need to upload a single image and then you just need to use "press to view your profile pic" button.Download both the images by using right-click of mouse and upload them on Facebook.Align the thumbnail of profile pic as it suits best to you.

My app utilizes the Html 5 Canvas and File API combined with some smart JavaScript. The inclusion of these elements makes sure that old browsers(read Internet Explorer) will not be a part of it.So use this app in the browsers which support HTML5 and JavaScript.


Any criticism or suggestions for the development of the app is always welcome.Mail me at aribachhawat@gmail.com for that.

I will explain the source code of the app in my next post.Till then HASTA la VISTA baby.