This week's book giveaway is in the OO, Patterns, UML and Refactoring forum. We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line! See this thread for details.
I have to read logs from a production Unix server and display them using a web-app. I have the ip-address and the file path.
I have two questions:
1) The log files are updated each minute and I have to read the latest logs, what would be the correct approach.
2) The current implementation reads logs from a netApp server, I read using HTTP from there. What is the best way to read from Unix server.
"One of the best things you could do is to simplify a larger application into a smaller one by reducing its process and complexity - Fowler"
You can possibly use something like the apache FTPClient to ftp the log file(s) from the unix server to your web app host. Depends on how fast your network is, how big the log files are, if ftp is enabled (among other things!).
log files get to 10-12 MB by the end of the day( one file each day ).
FTP is enabled, but do you want me to store it to localhost and then read from there ?
Joined: Jul 04, 2009
Well, that's one way of doing it... how often will your users be accessing it? Can you do something like pull the latest log file every five minutes or so, if you can't pull at every request? That way your users will be viewing data at most five minutes old. I don't know how acceptable this would be in your application, you have to decide that.
yeah, that is one way..
but the user can ask for the latest log.. i.e the event which has just been logged.
This gives me no time buffer.
Joined: Jul 04, 2009
I don't know of a mechanism that'll give you only the updated bits of a file on a remote host. How about deploying a transmitter application (a JMS client?) on your unix server that monitors the log file and publishes new events to your web app? Lots of possibilities there...
How about simply writing a syslog remote client (if not yet available in Java)?
In general every UNIX application is capable of logging to syslog, which is what your application maybe already does. If you use more modern syslog replacements like syslog-ng you can simply configure it to transmit log entries via ordinary socket connections to specific hosts. So you could delegate this problem to a syslog-ng server running on the box in question and listen for the log entries you get from it in your web app. How about this idea?
Joined: Jul 04, 2009
Agreed Marco, that is a better solution than having to deploy and manage 30 log transmitter instances.
I'll have to read more about syslog and see what is required for it.. can we deploy it on weblogic .. I mean where does it work ?
Joined: Jul 04, 2009
You don't have to deploy anything. It should already be running on your unix server as a daemon process. You configure it to publish your logs to a remote server (in this case, your app server). Your app server can then read the data off of the socket and do its thing with it.
1.) You would have to configure the application/application server to log to syslog (which can be done with a SyslogAppender for example if you're using log4j)
2.) On the server box you should configure syslog (or a modern version like syslog-ng) to filter the corresponding application logs and send them via network to your (remote) logging application.
3.) Remote syslog uses ordinary UDP socket connections, so you could rely on this on your "syslog client side". But I'm pretty sure there are ready-made libraries to receive the syslog messages in your logging application.
Syslog is the central point in this configuration!
The application that produces logs writes them to a local syslog server (e.g. via log4j) which is available on every UNIX box.
The local syslog service may then write these logs to a local log file and additionally "write" then to the network.
Your "log viewing web application" is then only one possible network receiver for the log message sent by the remote syslog (which is local to your application server).
Newer and more efficient variants of syslog may also support better network usage by using multicast etc. but the basic scenario should work for you! In fact it's the best idea in any clustered environment to aggregate logs at a central point. Then you know everytime where you have to look for logs of all your servers and you don't have to worry that any server may fill up its disk with unrotated/undeleted logs.
From a quick look log4j seems to provide a SocketAppender! This would of course be a very good alternative if you don't need the logs in syslog anyway. Unfortunately I haven't used this before so I can't tell you details but it should do pretty much the same I described above but without the extra step to involve a syslog server ;-)
Additional note from the log4j homepage! There are many useful appenders besides the well-know file or stdout appenders:
"The ability to selectively enable or disable logging requests based on their logger is only part of the picture. Log4j allows logging requests to print to multiple destinations. In log4j speak, an output destination is called an appender. Currently, appenders exist for the console, files, GUI components, remote socket servers, JMS, NT Event Loggers, and remote UNIX Syslog daemons. It is also possible to log asynchronously."
As I wrote in my last post you won't have to do anything UNIX specific if the application/application server which is producing the logs already uses log4j (or maybe a similar alternative). You just have to modify the log4j setting to use an additional appender which should send you the log entries via network (socket connection, JMS, etc.). If you can't even change this logging configuration, then what are you allowed to do on this server? I guess, anyone will have to provide any way to access these log files whatever this means :-)
You can also use an API like JSch to access/ftp the logs from UNIX servers.
Like you said you have 30 UNIX boxes, I too had the same scenario where I had to read certain attributes from different files on each Unix box and display them on the web app interface every15 min. I have built an architecture that does this via multi-threading. Just go through JSch once and see if you are interested.
Experience and talent are independent of age
I’ve looked at a lot of different solutions, and in my humble opinion Aspose is the way to go. Here’s the link: http://aspose.com