Is my problem because of 'StringBuilder'? Or do I need to some memory management (somehow)?

[Dave Wilmsmeyer]

Hi,

I am sorry if I put this in the wrong topic, but I am not really sure exactly where to put this issue.

So, a while back, I had asked (here) how to have my Java 1.6 program write updates into a JTextArea while it was executing processes, and I was pointed to SwingWorker. Here is a link to my previous topic . . .

https://coderanch.com/t/567777/GUI/java/show-new-text-JTextArea-while

And since then I have modified my code so that it now uses a SwingWorker to do all of the non-Swing tasks in a background thread. The SwingWorker reads a text file, searches for specific data, then does some processes on that data and then displays the results in HTML browsers. The files that the program will have to read are text, and they vary in size, 1KB - 900MB, but typically, the files range 1MB-100MB.

Here is my issue. The program works fine with any file size when I exectue it from Netbeans (in fact it is quite fast). However, if I execute the program outside of Netbeans, then on the LARGE files (>200MB), if fails to complete the processing. I do not get any errors of any type other that ones that I wrote into the code to let me know what happened. I also have the program do some logging, and I can see that the problem always occurs AFTER the program has read in ~200MB from a file.

The way the program works is it ... 1) Sets up a GUI 2) A user selects a file to process 3) Then the program creates a SwingWorker (background) thread and reads the file, does the processing, and then done() will write out the results.

Here is code that I have in the program that does the reading ...

           BufferedReader br = new BufferedReader(new FileReader(file));
           StringBuilder fileData = new StringBuilder();
           String line; 
           int loopCounter = 0;
           moreEvents = true;       
           while ((line = br.readLine()) != null) 
           {
             fileData.append(line);
             fileData.append("\n");
             if (fileData.length() >= 4000) 
             {
               loopCounter++;
               if (loopCounter%250 == 0) 
               {
                   Text.requestFocus();
                   Text.append("\nWorking ...");        
               }
               processData(fileData.toString());
               fileData.delete(0,fileData.length());
             }
           }   
           if (fileData.length() > 0)   
           {
              processData(fileData.toString());
           }
           moreEvents = false;

The reason I that I have it process after it has read 4K is for performance. That is, the program will read, and process 4K at a time. And then after it has read and processed all the 4K bundles, it puts all back into an ArrayList so that it can be all printed out.


So, I am still a greenhorn (obviously) and I am lost as to how to resolve this problem. As the program works just fine in Netbeans, I am thinking that my issue is possibly . . .

- A memory management issue? That is, Netbeans is doing some memory management so that the program works when it is executing there.

and/or

- Is the issue maybe because I am using StringBuilder?

Thanks in advance!

[Michael Dunn]

I can recall seeing this problem before, where? outcome? so I googled
jtextarea read large files

this might be worth a read

http://www.dreamincode.net/forums/topic/224430-reading-in-large-text-files1gb-10gb-java-swing/

[Darryl Burke]

Dave, when posting code please UseCodeTags <--link

I've added them for you this time.

[Dave Wilmsmeyer]

Sorry Darryl, from now on I will use CodeTags.

And, thanks Michael. However, I have a problem with using FileChannel as I don't know how to specify that it read in entire lines of text. That is, I need to have the entire lines of the input file intact because of the type of data that I am processing. But, I will try to see if there is any way to control the reads (using FileChannel) to function as a 'readLine' does.

One other thing, I have to correct my problem description. I was incorrect in my statement "the problem always occurs AFTER the program has read in ~200MB from a file" , the problem occurs after the program has read in ~20MB from a file. :-(

Anyways, I really would like to know how to set up my code to work as it does in NetBeans. Unfortunately, at this point, I still don't know exactly why my code works in NetBeans, but not outside of Netbeans.

[Dave Wilmsmeyer]

!! UPDATE !!

The problem is NOT occurring during the reading. The problem is occurring when the processed data is being put into a list.

The reason for my misunderstanding was because my error logging was not working as I expected it to (but that is another issue).

Anyways, the problem is occurring while the program is putting processed data strings in an ArrayList<String>, such as ...

resultList.add(tempList.get(x)) <- where tempList and resultList are both ArrayList<String>

... and after ~121,000 entries (this varies), the process stops (for some reason) long enough for done() to think the processing is finished and then it writes out the totals.

And although I now know exactly where the problem is occurring, I guess I still have the same questions, which are ...

WHY is this happening? That is, am I running out of memory? I guess I am going to have to learn how to use NetBean's Profiler so that I can see how the resources are being consumed by my code.

And, why does this code work in NetBeans, but not outside of NetBeans? That is, why will NetBeans continue to build the ArrayList without issue, but the ArrayList runs into a wall (of some sort) outside of NetBeans.

Anyways, sorry for the confusion, I am still learning here.

[Michael Dunn]

probably worth posting your problem in the netbeans forum (search first - might get lucky),
where you'll have many dedicated users/contributors who may have come across this issue before.

http://forums.netbeans.org/

[Dave Wilmsmeyer]

Another update , if I run the program from the command line, such as . . .

c:\java -jar "DecodeLogs.jar"

the program works again!?!?

I am really lost here now. I mean, I don't see how it could be a memory issue is it works like this.

So, as a test, I downloaded the newest version of Netbeans (7.1.1) and I then re-cleaned/re-built (for the 100th time), but still the same behavior. The code works fine in the IDE, but not outside the IDE.

Does someone have any ideas on how I can trouble shoot this?

I guess I will try some other forums.

[Michael Dunn]

> I guess I will try some other forums.

1) include a link to your topic here - people can then read the history here,
rather than you having to repeat it all.

2) if a solution comes up, please post the solution (or link to) here,
as this (general) problem does pop up occasionally.

[Dave Wilmsmeyer]

FYI, I tested my code in Eclipse (3.7.2) and I got the exact same behavior. That is, the code works fine while executing inside the IDE. But outside the IDE, it does not complete the processing of all of the data.

Again, the problem is not with the reading in of the data. The problem occurs while re-assembling the processed results into an ArrayList<String>. But, as I never did get any system generated errors nor do I get any code crashes, I was not ever able to pin down exactly why this behaved this way.

Anyways, this was taking too long to figure out and I have since gone back to the drawing board and did a design change. And, now everything works.

Thanks just the same Michael.