aspose file tools*
The moose likes Hadoop and the fly likes Log Analysis Using Hadoop Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Databases » Hadoop
Bookmark "Log Analysis Using Hadoop" Watch "Log Analysis Using Hadoop" New topic
Author

Log Analysis Using Hadoop

Srikanth Gaddam
Greenhorn

Joined: Mar 18, 2013
Posts: 1
Hi,

I am newbie to the Hadoop World though i am an experienced developer in Java. Doing a POC in Hadoop.
The requirement for the POC is to do a predictive analysis for the Apache/Any server logs using Hadoop and report the result in the dashboard.

Here are the steps i have done as of now.
A. Hadoop single node cluster environment is installed and configured properly.
B. Run few examples in the Hadoop Environment including WordCount.
C. Identified that every time i need to load the data into HDFS and get the result from the HDFS manually.

Here are my questions...
1. I want the full flow from Program run to the report display in Dashboard to be automated without using tools like Flume and Sqoop. If these tools are required then how?
Ex: if i run the LogAnalyser class in the Hadoop (the entire process should be automated) the report should be visible in the Dashboard.
2. I have multiple log files with different pattern used in the log writing in the log files. How to read multiple log files with different patterns in the single Driver class. Please give the Best practice.
3. Which is the best option for GUI to report the result in the Dashboard.
4. Please refer any resources related to this requirement ... so that i will go through them.

Thank you.
 
Gartner says :Bigdata will be most advanced analytics products by 2015 !

Time to Become Big data architect by learning Hadoop(Developer, Administration,Analyst,QA),Cassandra,MongoDb,HBase,Datascience, Mahout, Splunk,R etc) from scratch to expert level

https://intellipaat.com/course-cat/big-data/?utm_source=coderanch%20&utm_medium=text&utm_campaign=coderanchdx1
 
subject: Log Analysis Using Hadoop