aspose file tools*
The moose likes Java in General and the fly likes Reading logs from Unix server Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Reading logs from Unix server" Watch "Reading logs from Unix server" New topic
Author

Reading logs from Unix server

Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

Hi All,

I have to read logs from a production Unix server and display them using a web-app. I have the ip-address and the file path.
I have two questions:
1) The log files are updated each minute and I have to read the latest logs, what would be the correct approach.
2) The current implementation reads logs from a netApp server, I read using HTTP from there. What is the best way to read from Unix server.


Anrd
"One of the best things you could do is to simplify a larger application into a smaller one by reducing its process and complexity - Fowler"
Dipanjan Kailthya
Greenhorn

Joined: Jul 04, 2009
Posts: 20
You can possibly use something like the apache FTPClient to ftp the log file(s) from the unix server to your web app host. Depends on how fast your network is, how big the log files are, if ftp is enabled (among other things!).
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

log files get to 10-12 MB by the end of the day( one file each day ).
FTP is enabled, but do you want me to store it to localhost and then read from there ?
Dipanjan Kailthya
Greenhorn

Joined: Jul 04, 2009
Posts: 20
Well, that's one way of doing it... how often will your users be accessing it? Can you do something like pull the latest log file every five minutes or so, if you can't pull at every request? That way your users will be viewing data at most five minutes old. I don't know how acceptable this would be in your application, you have to decide that.

Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

yeah, that is one way..
but the user can ask for the latest log.. i.e the event which has just been logged.
This gives me no time buffer.
Dipanjan Kailthya
Greenhorn

Joined: Jul 04, 2009
Posts: 20
I don't know of a mechanism that'll give you only the updated bits of a file on a remote host. How about deploying a transmitter application (a JMS client?) on your unix server that monitors the log file and publishes new events to your web app? Lots of possibilities there...
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

nice idea..
let me elaborate the scenario.. we have around 30 unix servers containing logs for different data.
and I'll have to check if we can deploy something on them.

At present we FTP logs from them to a netApp location once a day, and the web-app reads from here.


Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1280

How about simply writing a syslog remote client (if not yet available in Java)?

In general every UNIX application is capable of logging to syslog, which is what your application maybe already does. If you use more modern syslog replacements like syslog-ng you can simply configure it to transmit log entries via ordinary socket connections to specific hosts. So you could delegate this problem to a syslog-ng server running on the box in question and listen for the log entries you get from it in your web app. How about this idea?

Marco
Dipanjan Kailthya
Greenhorn

Joined: Jul 04, 2009
Posts: 20
Agreed Marco, that is a better solution than having to deploy and manage 30 log transmitter instances.
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

I'll have to read more about syslog and see what is required for it.. can we deploy it on weblogic .. I mean where does it work ?
Dipanjan Kailthya
Greenhorn

Joined: Jul 04, 2009
Posts: 20
You don't have to deploy anything. It should already be running on your unix server as a daemon process. You configure it to publish your logs to a remote server (in this case, your app server). Your app server can then read the data off of the socket and do its thing with it.
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1280

1.) You would have to configure the application/application server to log to syslog (which can be done with a SyslogAppender for example if you're using log4j)

2.) On the server box you should configure syslog (or a modern version like syslog-ng) to filter the corresponding application logs and send them via network to your (remote) logging application.

3.) Remote syslog uses ordinary UDP socket connections, so you could rely on this on your "syslog client side". But I'm pretty sure there are ready-made libraries to receive the syslog messages in your logging application.

So it's basically "server app" --> "application server" --> "local syslog" --> "remote syslog-client/your logging application"


Marco
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

then I can write logs to netApp directly and my app will read from the same place.
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1280

Syslog is the central point in this configuration!

The application that produces logs writes them to a local syslog server (e.g. via log4j) which is available on every UNIX box.

The local syslog service may then write these logs to a local log file and additionally "write" then to the network.

Your "log viewing web application" is then only one possible network receiver for the log message sent by the remote syslog (which is local to your application server).

Newer and more efficient variants of syslog may also support better network usage by using multicast etc. but the basic scenario should work for you! In fact it's the best idea in any clustered environment to aggregate logs at a central point. Then you know everytime where you have to look for logs of all your servers and you don't have to worry that any server may fill up its disk with unrotated/undeleted logs.

Marco
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

Thanks Marco,
I am aware of log4j, but what if I can't change anything on the UNIX server.
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1280

From a quick look log4j seems to provide a SocketAppender! This would of course be a very good alternative if you don't need the logs in syslog anyway. Unfortunately I haven't used this before so I can't tell you details but it should do pretty much the same I described above but without the extra step to involve a syslog server ;-)

Additional note from the log4j homepage! There are many useful appenders besides the well-know file or stdout appenders:

"The ability to selectively enable or disable logging requests based on their logger is only part of the picture. Log4j allows logging requests to print to multiple destinations. In log4j speak, an output destination is called an appender. Currently, appenders exist for the console, files, GUI components, remote socket servers, JMS, NT Event Loggers, and remote UNIX Syslog daemons. It is also possible to log asynchronously."

Marco
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

Thanks Marco,
I am aware of log4j, but
Aniruddh Joshi wrote:what if I can't change anything on the UNIX server.
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1280

As I wrote in my last post you won't have to do anything UNIX specific if the application/application server which is producing the logs already uses log4j (or maybe a similar alternative). You just have to modify the log4j setting to use an additional appender which should send you the log entries via network (socket connection, JMS, etc.). If you can't even change this logging configuration, then what are you allowed to do on this server? I guess, anyone will have to provide any way to access these log files whatever this means :-)

Marco
Aniruddh Joshi
Ranch Hand

Joined: Jul 29, 2008
Posts: 275

Yes Marco,

I have to read files from there.. I ave a username and pwd which worked when I telnetted to the machine.
Your suggestion sounds perfect.. I'll try to implement it
Himanshu Kansal
Ranch Hand

Joined: Jul 05, 2009
Posts: 257
You can also use an API like JSch to access/ftp the logs from UNIX servers.
Like you said you have 30 UNIX boxes, I too had the same scenario where I had to read certain attributes from different files on each Unix box and display them on the web app interface every15 min. I have built an architecture that does this via multi-threading. Just go through JSch once and see if you are interested.


Experience and talent are independent of age
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
 
subject: Reading logs from Unix server