File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Hadoop and the fly likes Hadoop Single Node Setup Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » Hadoop
Bookmark "Hadoop Single Node Setup" Watch "Hadoop Single Node Setup" New topic
Author

Hadoop Single Node Setup

Lopez Mirinda
Greenhorn

Joined: Apr 11, 2007
Posts: 11
My boss has asked me to setup single node hadoop on windows and said that we would be using it to create some POC on how to use Big Data in this setup.
So, when I say single node haoop, is that the same as single node hadoop cluster?

Also, I have been browsing and found the following links.
http://www.cs.brandeis.edu/~rshaull/cs147a-fall-2008/hadoop-windows/
This link talks about using cygwin, java and any stable version of hadoop. This seems rather simple and easy.

http://developer.yahoo.com/hadoop/tutorial/module3.html#vm
This is more confusing, it says we need vmware player, copying the hadoop virtual machine etc etc.

Which one do I follow for my POC?
chris webster
Bartender

Joined: Mar 01, 2009
Posts: 1479
    
  11

I haven't used Hadoop, but I understand that it relies on Unix shell commands, so if you are on a Windows machine, then you either need a Unix-shell for your Windows machine (which is what Cygwin does), or you need a Linux virtual machine running inside your Windows machine (which is what VMWare does). Alternatively, you can either install Linux to dual-boot on your current machine before installing Hadoop, or find a machine that already has Linux on it and install Hadoop there instead.

I don't use Cygwin, so I don't know how you would install/run Hadoop with Cygwin, but I'm guessing you would open a Cygwin shell and then use Unix-style commands as indicated in the Hadoop instructions.

VirtualBox is an easy free alternative to VMWare - you can install VirtualBox on a Windows PC, then install a Linux VM to run inside VirtualBox e.g. here is a quick guide to installing Ubuntu Linux on VirtualBox. Once you've got Linux running inside VirtualBox, I guess you would log into your Linux VM and then follow the instructions for installing/running Hadoop on Linux.

If you're using Ubuntu, remember you may need to use "sudo" for running some commands that require extra permissions e.g. "sudo apt-get install ssh". Also, running a VM inside Windows takes extra memory, so make sure you have enough RAM to run both OS at the same time. If you dual-boot Linux instead, obviously this is not such a problem as you are either running Windows or Linux but not both at the same time.

FWIW, it may be slightly more work to install Linux (as a VM or dual boot) on your Windows PC, but once you've done this it's often much easier to get/install/use the Linux versions of many open source tools than the equivalent Windows versions.


No more Blub for me, thank you, Vicar.
Lopez Mirinda
Greenhorn

Joined: Apr 11, 2007
Posts: 11
thanks for the detailed info, chris.

I tried using cygwin for hadoop but as i see there are quite a few issues setting up hadoop.
So, thought of using this vmware and successful partly. though this is a good way to learn MapReduce, I would definitely try to install hadoop using cygwin as well.
Pablo Abbate
Ranch Hand

Joined: Aug 06, 2012
Posts: 30

Although this for linux, could be useful.

Cloudera Hadoop


Nissi Group, Posicionamiento Web, Desarrollo de Software, DiseƱo Web
Blog , LinkedIn
MadhaviLatha Polanki
Greenhorn

Joined: Dec 18, 2012
Posts: 4
Hi Lopez, could you please let me know if you have installed hadoop using cygwin succesfully. I have got few errors while formatting the name node in Hadoop. Could you please help me in resloving the issue. Thanks in advance.
Lopez Mirinda
Greenhorn

Joined: Apr 11, 2007
Posts: 11
madhavi,
please post the error, the hadoop version used. would be more helpful in debugging
MadhaviLatha Polanki
Greenhorn

Joined: Dec 18, 2012
Posts: 4
Hi Lopez, I will provide you with the screen shot for the error soon. Thank you.
MadhaviLatha Polanki
Greenhorn

Joined: Dec 18, 2012
Posts: 4
Hi Lopez. Sometimes I get an error regarding SSHD service start. Sometimes I get the error in running the local host and the error message for this is " Connection Refused". Now while extracting the hadoop jar file to the folder "C:\cygwin\user\local" folder I got the eror saying I do not have permissions to do this task. Could you please help me in knowing what might be the reason for this? I have attached the document for this error.


[Thumbnail for hadooperror.png]

Lopez Mirinda
Greenhorn

Joined: Apr 11, 2007
Posts: 11
try running cygwin terminal as "administrator" (on the terminal shortcut icon, right click, run as admin). and try to untar the hadoop distribution from the terminal.
MadhaviLatha Polanki
Greenhorn

Joined: Dec 18, 2012
Posts: 4
The problem is that I am running the cygwin as "Administrator user", but still I am encountering such problems.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Hadoop Single Node Setup
 
Similar Threads
Mapreduce using Java
requirements to install Hadoop on a notebook?
Error while installing Hadoop
what's Hadoop ?
Hadoop Clustering on Single machine