aspose file tools*
The moose likes I/O and Streams and the fly likes Is there any efficient and tested java util library that has a java version of the tail command? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Is there any efficient and tested java util library that has a java version of the tail command?" Watch "Is there any efficient and tested java util library that has a java version of the tail command?" New topic
Author

Is there any efficient and tested java util library that has a java version of the tail command?

Jimi Svedenholm
Ranch Hand

Joined: May 19, 2001
Posts: 53
Hi,

I'm wondering if anyone knows of any java util library (like Apache Commons) that has an efficient and tested java implementation of the unix tail command? By effective I mean that it should be able to handle very large files (>1GB), with various unknown line lengths, quickly without lots of I/O or high memory usage. And by tested I mean something that can safely be used in production environments. In short, something as safe and efficient (well almost at least) as the tail command on some unix OS.

The reason I'm asking this is because I want to be able to display a server log file in a administration webapplication, and since the log file can grow very large I only want to show the last N lines. But I was amazed to notice that I couldn't seem to find any "java tail" function. I found a lot of different code examples, but that doesn't help me much. This seems like such a low level function that it should really exist in a tested, efficient and hopefully nearly bug free java util library. Writing it myself (even if that means copy pasting example code from some forum) is way to error prone for my taste.

So, does anyone know of any such existing such java util library? Preferably one that exists in a standard maven repository.

Regards
/Jimi
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42289
    
  64
If it's a web app, what part do you envision Java to play? Streaming log entries to an HTML/JavaScript client? Or in an applet that reads from a streamed server source?


Ping & DNS - my free Android networking tools app
Jimi Svedenholm
Ranch Hand

Joined: May 19, 2001
Posts: 53
Ulf Dittmer wrote:If it's a web app, what part do you envision Java to play? Streaming log entries to an HTML/JavaScript client? Or in an applet that reads from a streamed server source?


It would simply be a page displaying the last say 1000 lines of the file, in a scrollable div, that would do an ajax refresh automatically every 30 seconds or so (with the possibility to turn that off using a check box on the side) and scroll the div to the end on each update.
James Sabre
Ranch Hand

Joined: Sep 07, 2004
Posts: 781

It may not be tested to the level you require but reply #10 of http://forums.sun.com/thread.jspa?threadID=709737.

Of course you could always write a test harness for yourself!


Retired horse trader.
 Note: double-underline links may be advertisements automatically added by this site and are probably not endorsed by me.
Jimi Svedenholm
Ranch Hand

Joined: May 19, 2001
Posts: 53
James Sabre wrote:It may not be tested to the level you require but reply #10 of http://forums.sun.com/thread.jspa?threadID=709737.

Of course you could always write a test harness for yourself!


Well... As I said, it is very easy to copy paste some example code from the net. But that is not what I am looking for. How do I know that anyone else is using this code currently? And can I get news of any bugs found? The most likely scenario is that if anyone uses that code and then later finds some bugs he simply tries to fix the bugs himself and I have no idea about it.

But the lack of response seems to indicate that what I am looking for doesn't exist. And I guess that the example code you link to is better then nothing. Although I just tested it, and it can't handle windows line breaks (CR + LF becomes LF + CR and that is interpreted as two separate line breaks), so I had to fix that. Also it doesn't handle UTF-8, but that I can live with I guess.

Thanks for the tip, even though I'm still a bit perplexed about the fact that I have to reinvent the wheel like this.

Regards
/Jimi
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18669
    
    8

Going back one step from your "tail" requirement to your original requirement of being able to display log entries from one application in a second application...

If I wanted to do that, here's what I would do:

  • Use log4j in the application doing the logging
  • Configure it to use a SocketAppender
  • Have the second application run a SocketServer to receive the logs


  • Now perhaps you can't do that, in which case just disregard the advice. But I have actually done that (in my case the second application actually needed to act on entries in the first application's logs.)
    James Sabre
    Ranch Hand

    Joined: Sep 07, 2004
    Posts: 781


    Well... As I said, it is very easy to copy paste some example code from the net. But that is not what I am looking for.

    Which is why I was slow pointing you at it.


    How do I know that anyone else is using this code currently?

    You don't but just because someone may be using it does not make it perfect. I needed an ASCII85 encoder for a project and quickly turned up one from FREEHEP. It was rubbish. I created a simple JUnit test harness which quickly found 3 show stopper bugs. I reported the bugs to FREEHEP and a long time later I got some feedback saying that they had fixed at least two of the bugs but I have lost confidence in the whole FREEHEP site. While they were fixing a couple of bugs I wrote one together with a comprehensive JUnit tests. I have confidence in my code.

    The moral of the story is - test for yourself.


    And can I get news of any bugs found?

    Most probably not unless someone posts to that thread. You can set yourself up to receive an email if anyone posts to the thread but ...


    The most likely scenario is that if anyone uses that code and then later finds some bugs he simply tries to fix the bugs himself and I have no idea about it.

    Of course.


    But the lack of response seems to indicate that what I am looking for doesn't exist. And I guess that the example code you link to is better then nothing. Although I just tested it, and it can't handle windows line breaks (CR + LF becomes LF + CR and that is interpreted as two separate line breaks), so I had to fix that. Also it doesn't handle UTF-8, but that I can live with I guess.

    I had to produce that code in the first place because, like you, I needed tail() type functionality for a home project for looking at log files. I was primarily using iso-8859-1 encoding and I think I said in the thread that it did not work for multi-byte character sets. I have moved on since then. I developed an improved version for a customer that handles the Java mandated character sets of US-ASCII, ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE and UTF-16. It actually handles all the iso-8859-x character sets and may handle others by accident. The only character sets it is known to fail to handle is EBCIDIC family though there may be others. It deals with the CR-LF problem.

    The new improved version is quicker than the original and the heart of the code is actually much smaller than the original. I cannot release it since it is owned by my customer. One day I might get round to creating a more general version and release that but this won't satisfy your requirement for a significant user base and well defined bug reporting system.


    Thanks for the tip, even though I'm still a bit perplexed about the fact that I have to reinvent the wheel like this
    .
    Why perplexed? Surely this sort of deficiency is what drives the Jakarta projects.
    Jimi Svedenholm
    Ranch Hand

    Joined: May 19, 2001
    Posts: 53
    Paul Clapham wrote:Going back one step from your "tail" requirement to your original requirement of being able to display log entries from one application in a second application...

    If I wanted to do that, here's what I would do:

  • Use log4j in the application doing the logging
  • Configure it to use a SocketAppender
  • Have the second application run a SocketServer to receive the logs


  • Now perhaps you can't do that, in which case just disregard the advice. But I have actually done that (in my case the second application actually needed to act on entries in the first application's logs.)


    I actually had something like this in mind before, using some log4j appender to get access to the information, but when I asked in the Log4j user mailing list I only got one reply and involved a JdbcAppender. But I'm not very keen on causing more database traffic, so that is not an alternative for me. A SocketAppender sounds quite interesting, but in my case even a simple "InMemoryAppender" is enough, since the logging happens in the same webapplication.

    I actually got inspired by your suggestion, and decided to try and write my own appender, and it was quite simple. Sure, it doesn't give me access to events logged before a restart/crash, but that is a minor issue and I now provide a download link so the user can access the complete log file if needed.
    Rob Spoor
    Sheriff

    Joined: Oct 27, 2005
    Posts: 19722
        
      20

    Jimi Svedenholm wrote:I actually got inspired by your suggestion, and decided to try and write my own appender, and it was quite simple.

    Nice, isn't it?

    Sure, it doesn't give me access to events logged before a restart/crash, but that is a minor issue

    You could try parsing the existing file when the appender is created for an initial list of events. Afterwards you simply toss those away as your list gets more and more events.


    SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
    How To Ask Questions How To Answer Questions
    Jimi Svedenholm
    Ranch Hand

    Joined: May 19, 2001
    Posts: 53
    James Sabre wrote:
    I had to produce that code in the first place because, like you, I needed tail() type functionality for a home project for looking at log files. I was primarily using iso-8859-1 encoding and I think I said in the thread that it did not work for multi-byte character sets. I have moved on since then. I developed an improved version for a customer that handles the Java mandated character sets of US-ASCII, ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE and UTF-16. It actually handles all the iso-8859-x character sets and may handle others by accident. The only character sets it is known to fail to handle is EBCIDIC family though there may be others. It deals with the CR-LF problem.

    The new improved version is quicker than the original and the heart of the code is actually much smaller than the original. I cannot release it since it is owned by my customer.


    ah, too bad. Code like that really belongs in a apache commons project or similar.


    Why perplexed? Surely this sort of deficiency is what drives the Jakarta projects.


    Well, yes, but I really though that this deficiency had been noticed enough times before that it already existed in such a project, just that I hadn't found it.
    Jimi Svedenholm
    Ranch Hand

    Joined: May 19, 2001
    Posts: 53
    Rob Prime wrote:
    Jimi Svedenholm wrote:
    Sure, it doesn't give me access to events logged before a restart/crash, but that is a minor issue

    You could try parsing the existing file when the appender is created for an initial list of events. Afterwards you simply toss those away as your list gets more and more events.


    Yes, that actually sounds like a very nice compromise. Why didn't I think of that?

    The only thing is that I noticed a problem with log4j on windows. If I read the from the file at the same time as the DailyRollingFileAppender decides its time to rename the file there can be problems. Because, on windows a file rename fails if another process/thread has the file open. And apparently Log4j doesn't handle that properly, because when that happend to me yesterday, on my development machine, log4j decided to erase everything logged the previous day, and that is totally unacceptable if you ask me. It should really be able to handle that situation (maybe by trying to rename the file again, a few seconds later, but as a last resort at least keep the old log lines intact and just keep appending to the file).

    I guess I could do a OS check, and only read from the file on non-windows machines. But now it's midsummer weekend, so no more work for mefor a few days.

    Thanks for your suggestions, Rob.
     
    I agree. Here's the link: http://aspose.com/file-tools
     
    subject: Is there any efficient and tested java util library that has a java version of the tail command?