Win a copy of Svelte and Sapper in Action this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Paul Clapham
  • Bear Bibeault
  • Junilu Lacar
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Henry Wong
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • salvin francis
  • Frits Walraven
Bartenders:
  • Scott Selikoff
  • Piet Souris
  • Carey Brown

NodeManager NameNode ResourceManager DataNode SecondaryNameNode not running

 
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i followed this link to install hadoop http://thepowerofdata.io/setting-up-a-apache-hadoop-2-7-single-node-on-ubuntu-14-04/
at the end when i run
the response i get is
while according to the link i should be getting

i also tried to run the daemons manually by commands

but i get [fatal error] like this


please help
 
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Did you copy and paste contents of core-site.xml and other .xml files from that site as is? Your xml files shouldn't contain lines like "...". Those are in the article just to show that the file can have other configuration properties before or after the one shown.
 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


this is my core-site.xml.
so i need to remove '...'
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes.
Also, "fs.default.name" was deprecated in 2.x in favour of "fs.defaultFS". Deprecated names usually work, but there are no guarantees that every version will support them going forward. It's recommended to not use deprecated names.
 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
so what do i need to write instead of

after removing '..' i ran
but when i ran the response i am getting is

 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

so what do i need to write instead of fs.default.name


Read my previous reply again. And see the link given.

Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId:
file:/usr/local/lib/hadoop-2.7.0/etc/hadoop/core-site.xml; lineNumber: 21; columnNumber: 2;


So, open core-site.xml in a text editor and see what character is at line #21 column #2.
 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i changed "fs.defaultFS" but yet it was showing errors at some line number. i gone through the files in editor where i found out that i have used the tag
twice in cases of all 4 .xml files (core-site,mapred-site,yarn-site,hdfs-site)(as i had just copy pasted from online source)
after removing it i ran the commands

and finally i got the required response

 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks a lot Mr. Karthik Shiraly for efforts and patience. it was great help.
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Good to know it's working now

I'll close this thread as resolved.
 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
now that it's successfully installed, what steps would you suggest me to learn hadoop. i am sure by now you know my level in coding. which is poor. so please suggest me some path to understand and learn hadoop. i want to start my career in hadoop as soon as possible and i am ready to give 6-8hours a daily for it.
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

i want to start my career in hadoop as soon as possible and i am ready to give 6-8hours daily for it.


Reality:
I gather you're a fresher or recently out of college, yes? When I google "hadoop jobs", I don't see anything for freshers. Even the most minimal experience required was 2+ years.
So you are going up against the reality of how companies hire for these technologies.

The reason is simple.
For companies, these tools - hadoop/spark/storm, etc - are mere tools. Just glorified excel sheets, but on a massive scale, to extract useful business information out of their data.
If they could use excel, they'd definitely prefer that. But sometimes excel is too puny, and they have to resort to these big hammers.
Ultimately, they really want from tools is to derive maximum business value using them, and the fastest way is to hire people already
proficient and experienced in writing data mining algorithms using these tools.
Extracting information out of business data is a difficult problem on its own.
They don't want to waste time or money training employees from the ground up for this.

Required skills:
So at any interview, they'll start by testing your fundamentals of data structures, algorithm analysis, java/python, databases, SQL, and popular data formats like XML or JSON.
It also helps to understand some basics of statistics, because the quality of data as well as results of data mining are judged statistically.
These are the most fundamental skills necessary to write distributed data mining algorithms.
If you don't do well in them, they are unlikely to proceed to asking questions on technologies like hadoop, and they won't believe anything else you say.
Much practical knowledge in these areas can be picked up by simply working as a developer.

So before learning hadoop or whatever...

Learn your data structures and algorithm fundamentals.

Learn java better. Learn how to write code for basic data processing tasks like parsing and processing text files, XML and JSON.

Learn to use databases and SQL.

A lot of data science is actually done using python, and higher level technologies like Pig and Hive that build upon Hadoop.
Learn to write hadoop jobs in java, python, pig and hive.

Of course, to write these jobs, first you need problems to solve.
So think up your own data science problems to solve, or use something like Kaggle, and build a portfolio of them on Github or your own site.
They could be something very personal - like processing your own electricity bills.
Or you could try many of the public datasets available (check out data.gov.in for Indian open datasets).

Roadmap:
Research some small and mid-sized companies that have teams doing data science and data engineering, and is hiring freshers, but not necessarily for those teams.
Try and get hired in them. If you know your fundamentals well, small or mid sized companies won't reject you summarily for not having required experience.

Later, with some experience, you can try shifting to their data team. Even if that doesn't work out, you'd have built up enough skills to jump to another company's data team.
There are 2 kinds of data analysis - "exploratory" (where you extract information from available data), and "predictive" (where you predict what will happen from available data).
The latter comes in the realm of machine learning, and would be somewhat too advanced for you at this point. So concentrate on exploratory data analysis for now.
When you get hired as a data engineer, you'll anyway eventually run into predictive analysis at some point and you can pick up that skill then.

Good luck!
 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks for helping me understand the scenario in IT for fresher.
being a fresher, would it be helpful for me if i clear hadoop certification as well as OCA(java crtification) .
 
Karthik Shiraly
Bartender
Posts: 1210
25
Android Python PHP C++ Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

akshay naidu wrote:being a fresher, would it be helpful for me if i clear hadoop certification as well as OCA(java certification) .



I doubt including any of these certifications in your resume makes it stand out in the minds of HR or even technical interviewers in India. It certainly won't hurt, but it's unlikely to help. It's my personal opinion.

But from a knowledge improvement point of view, probably yes, if you have found that you learn things better when studying for certifications.
 
akshay naidu
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks for all the advices.
 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hey,
This is phanindra
I have same query in hadoop
when i am excuting map reduce in hive it is getting struck their with out going further execution
could you please help me to solve this and one more is secondary node is not showing in jps.
Below is the following error.
Thanks,
phanindra

#######################################################
hive> select count(*) from retaildata;

Query ID = hadoopuser_20200426001439_6b21ac08-873d-417c-a687-8772d3d6ab34

Total jobs = 1

Launching Job 1 out of 1

Number of reduce tasks determined at compile time: 1

In order to change the average load for a reducer (in bytes):

set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:

set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:

set mapreduce.job.reduces=<number>

Starting Job = job_1587837430446_0001, Tracking URL = http://phani-VirtualBox:8088/proxy/application_1587837430446_0001/

Kill Command = /usr/local/hadoop/bin/mapred job -kill job_1587837430446_0001
 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
A single node cluster setup is done in my windows machine. While starting hadoop daemons, I can see namenode and datanode is up and running but resorcemanager and nodemanager throws a shutdown msg. can anyone help to resolve the issue.
 
Slideshow boring ... losing consciousness ... just gonna take a quick nap on this tiny ad ...
the value of filler advertising in 2020
https://coderanch.com/t/730886/filler-advertising
    Bookmark Topic Watch Topic
  • New Topic