I am sorry if I put this in the wrong topic, but I am not really sure exactly where to put this issue.
So, a while back, I had asked (here) how to have my Java 1.6 program write updates into a JTextArea while it was executing processes, and I was pointed to SwingWorker. Here is a link to my previous topic . . .
And since then I have modified my code so that it now uses a SwingWorker to do all of the non-Swing tasks in a background thread. The SwingWorker reads a text file, searches for specific data, then does some processes on that data and then displays the results in HTML browsers. The files that the program will have to read are text, and they vary in size, 1KB - 900MB, but typically, the files range 1MB-100MB.
Here is my issue. The program works fine with any file size when I exectue it from Netbeans (in fact it is quite fast). However, if I execute the program outside of Netbeans, then on the LARGE files (>200MB), if fails to complete the processing. I do not get any errors of any type other that ones that I wrote into the code to let me know what happened. I also have the program do some logging, and I can see that the problem always occurs AFTER the program has read in ~200MB from a file.
The way the program works is it ... 1) Sets up a GUI 2) A user selects a file to process 3) Then the program creates a SwingWorker (background) thread and reads the file, does the processing, and then done() will write out the results.
Here is code that I have in the program that does the reading ...
The reason I that I have it process after it has read 4K is for performance. That is, the program will read, and process 4K at a time. And then after it has read and processed all the 4K bundles, it puts all back into an ArrayList so that it can be all printed out.
So, I am still a greenhorn (obviously) and I am lost as to how to resolve this problem. As the program works just fine in Netbeans, I am thinking that my issue is possibly . . .
- A memory management issue? That is, Netbeans is doing some memory management so that the program works when it is executing there.
- Is the issue maybe because I am using StringBuilder?
And, thanks Michael. However, I have a problem with using FileChannel as I don't know how to specify that it read in entire lines of text. That is, I need to have the entire lines of the input file intact because of the type of data that I am processing. But, I will try to see if there is any way to control the reads (using FileChannel) to function as a 'readLine' does.
One other thing, I have to correct my problem description. I was incorrect in my statement "the problem always occurs AFTER the program has read in ~200MB from a file" , the problem occurs after the program has read in ~20MB from a file. :-(
Anyways, I really would like to know how to set up my code to work as it does in NetBeans. Unfortunately, at this point, I still don't know exactly why my code works in NetBeans, but not outside of Netbeans.
The problem is NOT occurring during the reading. The problem is occurring when the processed data is being put into a list.
The reason for my misunderstanding was because my error logging was not working as I expected it to (but that is another issue).
Anyways, the problem is occurring while the program is putting processed data strings in an ArrayList<String>, such as ...
resultList.add(tempList.get(x)) <- where tempList and resultList are both ArrayList<String>
... and after ~121,000 entries (this varies), the process stops (for some reason) long enough for done() to think the processing is finished and then it writes out the totals.
And although I now know exactly where the problem is occurring, I guess I still have the same questions, which are ...
WHY is this happening? That is, am I running out of memory? I guess I am going to have to learn how to use NetBean's Profiler so that I can see how the resources are being consumed by my code.
And, why does this code work in NetBeans, but not outside of NetBeans? That is, why will NetBeans continue to build the ArrayList without issue, but the ArrayList runs into a wall (of some sort) outside of NetBeans.
Anyways, sorry for the confusion, I am still learning here.
Another update , if I run the program from the command line, such as . . .
c:\java -jar "DecodeLogs.jar"
the program works again!?!?
I am really lost here now. I mean, I don't see how it could be a memory issue is it works like this.
So, as a test, I downloaded the newest version of Netbeans (7.1.1) and I then re-cleaned/re-built (for the 100th time), but still the same behavior. The code works fine in the IDE, but not outside the IDE.
Does someone have any ideas on how I can trouble shoot this?
FYI, I tested my code in Eclipse (3.7.2) and I got the exact same behavior. That is, the code works fine while executing inside the IDE. But outside the IDE, it does not complete the processing of all of the data.
Again, the problem is not with the reading in of the data. The problem occurs while re-assembling the processed results into an ArrayList<String>. But, as I never did get any system generated errors nor do I get any code crashes, I was not ever able to pin down exactly why this behaved this way.
Anyways, this was taking too long to figure out and I have since gone back to the drawing board and did a design change. And, now everything works.
Thanks just the same Michael.
pie. tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop