• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Reading logs from Unix server

 
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

I have to read logs from a production Unix server and display them using a web-app. I have the ip-address and the file path.
I have two questions:
1) The log files are updated each minute and I have to read the latest logs, what would be the correct approach.
2) The current implementation reads logs from a netApp server, I read using HTTP from there. What is the best way to read from Unix server.

 
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You can possibly use something like the apache FTPClient to ftp the log file(s) from the unix server to your web app host. Depends on how fast your network is, how big the log files are, if ftp is enabled (among other things!).
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
log files get to 10-12 MB by the end of the day( one file each day ).
FTP is enabled, but do you want me to store it to localhost and then read from there ?
 
Dipanjan Kailthya
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, that's one way of doing it... how often will your users be accessing it? Can you do something like pull the latest log file every five minutes or so, if you can't pull at every request? That way your users will be viewing data at most five minutes old. I don't know how acceptable this would be in your application, you have to decide that.

 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
yeah, that is one way..
but the user can ask for the latest log.. i.e the event which has just been logged.
This gives me no time buffer.
 
Dipanjan Kailthya
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't know of a mechanism that'll give you only the updated bits of a file on a remote host. How about deploying a transmitter application (a JMS client?) on your unix server that monitors the log file and publishes new events to your web app? Lots of possibilities there...
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
nice idea..
let me elaborate the scenario.. we have around 30 unix servers containing logs for different data.
and I'll have to check if we can deploy something on them.

At present we FTP logs from them to a netApp location once a day, and the web-app reads from here.


 
best scout
Posts: 1294
Scala IntelliJ IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
How about simply writing a syslog remote client (if not yet available in Java)?

In general every UNIX application is capable of logging to syslog, which is what your application maybe already does. If you use more modern syslog replacements like syslog-ng you can simply configure it to transmit log entries via ordinary socket connections to specific hosts. So you could delegate this problem to a syslog-ng server running on the box in question and listen for the log entries you get from it in your web app. How about this idea?

Marco
 
Dipanjan Kailthya
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Agreed Marco, that is a better solution than having to deploy and manage 30 log transmitter instances.
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'll have to read more about syslog and see what is required for it.. can we deploy it on weblogic .. I mean where does it work ?
 
Dipanjan Kailthya
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You don't have to deploy anything. It should already be running on your unix server as a daemon process. You configure it to publish your logs to a remote server (in this case, your app server). Your app server can then read the data off of the socket and do its thing with it.
 
Marco Ehrentreich
best scout
Posts: 1294
Scala IntelliJ IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
1.) You would have to configure the application/application server to log to syslog (which can be done with a SyslogAppender for example if you're using log4j)

2.) On the server box you should configure syslog (or a modern version like syslog-ng) to filter the corresponding application logs and send them via network to your (remote) logging application.

3.) Remote syslog uses ordinary UDP socket connections, so you could rely on this on your "syslog client side". But I'm pretty sure there are ready-made libraries to receive the syslog messages in your logging application.

So it's basically "server app" --> "application server" --> "local syslog" --> "remote syslog-client/your logging application"


Marco
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
then I can write logs to netApp directly and my app will read from the same place.
 
Marco Ehrentreich
best scout
Posts: 1294
Scala IntelliJ IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Syslog is the central point in this configuration!

The application that produces logs writes them to a local syslog server (e.g. via log4j) which is available on every UNIX box.

The local syslog service may then write these logs to a local log file and additionally "write" then to the network.

Your "log viewing web application" is then only one possible network receiver for the log message sent by the remote syslog (which is local to your application server).

Newer and more efficient variants of syslog may also support better network usage by using multicast etc. but the basic scenario should work for you! In fact it's the best idea in any clustered environment to aggregate logs at a central point. Then you know everytime where you have to look for logs of all your servers and you don't have to worry that any server may fill up its disk with unrotated/undeleted logs.

Marco
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Marco,
I am aware of log4j, but what if I can't change anything on the UNIX server.
 
Marco Ehrentreich
best scout
Posts: 1294
Scala IntelliJ IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
From a quick look log4j seems to provide a SocketAppender! This would of course be a very good alternative if you don't need the logs in syslog anyway. Unfortunately I haven't used this before so I can't tell you details but it should do pretty much the same I described above but without the extra step to involve a syslog server ;-)

Additional note from the log4j homepage! There are many useful appenders besides the well-know file or stdout appenders:

"The ability to selectively enable or disable logging requests based on their logger is only part of the picture. Log4j allows logging requests to print to multiple destinations. In log4j speak, an output destination is called an appender. Currently, appenders exist for the console, files, GUI components, remote socket servers, JMS, NT Event Loggers, and remote UNIX Syslog daemons. It is also possible to log asynchronously."

Marco
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Marco,
I am aware of log4j, but

Aniruddh Joshi wrote:what if I can't change anything on the UNIX server.

 
Marco Ehrentreich
best scout
Posts: 1294
Scala IntelliJ IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
As I wrote in my last post you won't have to do anything UNIX specific if the application/application server which is producing the logs already uses log4j (or maybe a similar alternative). You just have to modify the log4j setting to use an additional appender which should send you the log entries via network (socket connection, JMS, etc.). If you can't even change this logging configuration, then what are you allowed to do on this server? I guess, anyone will have to provide any way to access these log files whatever this means :-)

Marco
 
Aniruddh Joshi
Ranch Foreman
Posts: 275
jQuery Eclipse IDE Spring
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes Marco,

I have to read files from there.. I ave a username and pwd which worked when I telnetted to the machine.
Your suggestion sounds perfect.. I'll try to implement it
 
Ranch Hand
Posts: 257
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You can also use an API like JSch to access/ftp the logs from UNIX servers.
Like you said you have 30 UNIX boxes, I too had the same scenario where I had to read certain attributes from different files on each Unix box and display them on the web app interface every15 min. I have built an architecture that does this via multi-threading. Just go through JSch once and see if you are interested.
 
reply
    Bookmark Topic Watch Topic
  • New Topic