aspose file tools*
The moose likes Java in General and the fly likes memory/cpu issue of reading/writing files Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "memory/cpu issue of reading/writing files" Watch "memory/cpu issue of reading/writing files" New topic
Author

memory/cpu issue of reading/writing files

david arnold
Ranch Hand

Joined: Apr 10, 2009
Posts: 133
Hi,

I am doing a project for monitoring the log files. When application(tomcat) start, it start one thread for one monitored file. The thread will run once very few seconds(pause by sleeping). Each time it will open and read the log file(bufferedReader) from the end line of previous reading. if any pre-defined keyword found in the log file, the matched line will be stored to stringbuilder, once it finish reading or the stringbuilder is bigger than a size, it will open particular file and write mathced content to the file.

The issue is that the (daily) log file will grow to be big (hundred Megebyte) from morning to the evening, and I noticed the cpu is 100% running when the file is big. I am wondering if there is good way to do this task like Bare Tail does which do not have much affect on memory and cpu?

Thanks.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12823
    
    5
Each time it will open and read the log file(bufferedReader) from the end line of previous reading.


Exactly how are you positioning at the end line of previous reading?

That is not something possible with BufferedReader which would have to read the whole file, resulting in what you observe.

Also note that a Reader is going to be doing character by character conversion to UNICODE, an expensive operation.

Bill
Anayonkar Shivalkar
Bartender

Joined: Dec 08, 2010
Posts: 1510
    
    5

david arnold wrote:Each time it will open and read the log file(bufferedReader) from the end line of previous reading.

Instead of just opening a file after a time interval, I would suggest something like checking last modified time. If file is not modified at all, there's no point in opening the file in first place.
david arnold wrote:if any pre-defined keyword found in the log file, the matched line will be stored to stringbuilder

I'm not sure which OS you are working with, but in Linux, this can simply be done by grep command. No need to even open a file using Java classes. On Windows, there might be something similar to it.

I hope this helps.


Regards,
Anayonkar Shivalkar (SCJP, SCWCD, OCMJD, OCEEJBD)
david arnold
Ranch Hand

Joined: Apr 10, 2009
Posts: 133
Thank you William and Anayonkar.

Exactly how are you positioning at the end line of previous reading?

That is not something possible with BufferedReader which would have to read the whole file, resulting in what you observe.


Since I used bufferedReader to read line by line, I have counter to remember the line number.

Also note that a Reader is going to be doing character by character conversion to UNICODE, an expensive operation.



Thank you. I did not know it, but I was wondering maybe it will be more efficient if I use stream. The reason i use bufferedReader is that it help to maitain the line by line format, when a line is read and it has keyword inside, it will be recorded with "\n" to the file, later it will be displayed on a browser and may need to send to email as well. so i do not know if i use stream, it will still easy to have this line format maintained.

Instead of just opening a file after a time interval, I would suggest something like checking last modified time. If file is not modified at all, there's no point in opening the file in first place.


Good point, i will add that.

I'm not sure which OS you are working with, but in Linux, this can simply be done by grep command. No need to even open a file using Java classes. On Windows, there might be something similar to it.


It is windows, it will be wonderful if it has such feature, but I am wondering if there is any.

Thanks again for the help!

Regards,
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18991
    
  40

david arnold wrote:
Exactly how are you positioning at the end line of previous reading?

That is not something possible with BufferedReader which would have to read the whole file, resulting in what you observe.


Since I used bufferedReader to read line by line, I have counter to remember the line number.



You can try the java.io.RandomAccessFile class, which can be used to seek to the point that you want, without having to read everything all over again.

Henry

Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
david arnold
Ranch Hand

Joined: Apr 10, 2009
Posts: 133
You can try the java.io.RandomAccessFile class, which can be used to seek to the point that you want, without having to read everything all over again.


Thank you Henry. I did think about using the RandomAccessFile before, i will try that.

Currently, I open the file and read and close it every few seconds, I am wondering if I should open the file, then have a loop checking for new data coming in without close it. I am not sure which way is better?

Thanks.
Carlo Moore
Greenhorn

Joined: Aug 02, 2005
Posts: 27
I wonder if that will work, I thought that when you opened a java.io.File for reading a snapshot of the file is opened in memory.

Edit: Maybe the line that's read is the only thing stored in memory and not the file itself.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: memory/cpu issue of reading/writing files