My problem:
I have an input textfile of 1000 records (1000 lines) consisting of birthdate, personal id number and geographic location number.
I am to produce an output textfile by joining the input records with data in a database table (Sybase ASE 12.5 database, I connect to it with
JDBC driver: JConnect5) of 5 000 000 rows plus (most or all of the input data is found in this table) and for each of the input records add four more records with identical geographic location number and year of birth. The output then is to consist of about 5000 records (lines).
I use JBuilder5 Enterprise.
Something that I have tried:
Use java.sql.*
Get a resultset from the database table
Statement statement = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
ResultSet rs;
rs = statement.executeQuery(�select datepart(year, birthdate), datepart(month, birthdate), datepart(day, birthdate), personalid, locationid from <table> order by locationid, birthdate�);
This operation takes about nine minutes.
After a while of traversing the ResultSet an OutOfMemoryError is thrown.
I have outputted some memory values, by using Runtime methods totalMemory() and freeMemory(), during various stages:
Memory used after allocated rs: 1800520
Free memory after allocated rs: 718160
Memory used after processing two of the inputdata: 14874960
Memory used after processing four of the inputdata: 16731768
Memory used after processing six of the inputdata: 57160416
Then before eight inputdata have been processed:
Exception occurred during event dispatching:
java.lang.OutOfMemoryError
<<no stack trace available>>
I would very much appreciate tips and help to any possible solutions to this !
Any explanations to the workings of ResultSet, that has to do with this problem.
Ways to work around this problem.
Any other tips, solutions.