• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Heap Dump

 
Ranch Hand
Posts: 138
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi all,

Well we have severe problems facing with heap dump.

One of the reason is that while huge data exporting to excel.

Probably we are fetch that data/records from database and storing into string array. Like daily 11000 to 15000 records are fetching and storing in string array.
After that hole string object is storing in a hash table.
Due to this the size of the object is bulging and at some point the server is crushing creating heap dumps.

Please help me how to resolve his probelm.

 
Saloon Keeper
Posts: 7585
176
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The obvious approach would be to either increase the memory allocation, or not to store so many objects. Is there are a need to store all those objects in memory? Can't the file be created by progressively getting only a few objects from the DB, working on those, and then releasing them?
 
naga eswar
Ranch Hand
Posts: 138
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks TIM.

Exactly I am also thinking on same logic. I want divide the data into chunks means bringing 1000 or 2000 records per one time and transfer one by one into excel. I am thinking about the reusability of object. Is this works...???

My application flow: JSP/SERVLET --> CONTROLLER --> ACTION --> SESSION FACADE --> SESSION BEAN --> ENTITY BEAN --> DAX --> DATABASE

there is directly flow from SESSION BEAN to DAX also.

The logic of executng query, fetching data and storing in objects is developed in DAX. I am fighting for the logic.
Below is code in DAX.... code for transfering the data into excel is developed in JSP.







 
Tim Moores
Saloon Keeper
Posts: 7585
176
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm not sure what you're asking; are you stuck implementing a solution where data is retrieved and worked on piecemeal? If so, where?

code for transfering the data into excel is developed in JSP.


That's bad design. You should move that code to a servlet or backing bean; it has no place in a JSP.
 
naga eswar
Ranch Hand
Posts: 138
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes Tim.... really bad code.


I will bring it back to servlet. Here my problem is that how to append the 1000, 1000 records to the already existing excel. means first it fetches 1 to 1000 records from database and inserts to excel and again it comes back fetches 1000 to 2000 records and should append from row 1001 in the existing excel.

For a while below is the code in jsp transfering data to excel.

Hashtable htDataToExport = null;

try{
response.setHeader("Content-Disposition","attachment; filename=\"DetailReport.xls\"");
htDataToExport = (Hashtable)request.getAttribute("ResultObject");
String strException = (String)request.getAttribute("EXCEPTION");
int iFlag = 1;
String strDtFrom = "";

%>
<HTML>
<title> Details Report</title>

<SCRIPT>
function init()
{
<%
if(strException != null)
{
%>
alert('<%= strException %>');
<%
}
%>
return false;
}
</SCRIPT>
<BODY onLoad="init();">
<form name="frmMain">
<%
if(htDataToExport != null && htDataToExport.size() > 0){
strDtFrom = (String)htDataToExport.get("fromDate");
}
%>
<TABLE>
<TR>
<TD>
<B>Detailed Collection </B>
</TD>
<TD ALIGN="RIGHT">
<B>Date : </B>
</TD>
<TD>
<B><%=JspUtil.checkNull(strDtFrom)%> </B>
</TD>

</TR>
<TR></TR>
</TABLE>
<table border=1 cellpadding=0 cellspacing=0 width="100%">
<TR>
<TD><B>Depositing Location</B></TD>
<TD><B>Sun GL Code</B></TD>
<TD><B>Instrument Category</B></TD>
<TD><B>Instrument Type</B></TD>
<TD><B>Instrument Creation ID</B></TD>
<TD><B>Instrument Creation Date</B></TD>
<TD><B>Instrument Receipt Date</B></TD>
<TD><B>Collection Deleted ? (Y/N)</B></TD>
<TD><B>Policy Number</B></TD>
<TD><B>Instrument Number</B></TD>
<TD><B>Instrument Date</B></TD>
<TD><B>MICR Code</B></TD>
<TD><B>Bank Name</B></TD>
<TD><B>Bank Branch</B></TD>
<TD><B>Clearance Type</B></TD>
<TD><B>Instrument Amount</B></TD>
<TD><B>PA Posted (Y/N)?</B></TD>
<TD><B>Collection Rejected (Y/N)?</B></TD>
<TD><B>PA Number</B></TD>
<TD><B>PA Date</B></TD>
<TD><B>RTF Number</B></TD>
<TD><B>RTF Date</B></TD>
<TD><B>RTF Posted (Y/N) ?</B></TD>
<TD><B>RTF Rejected (Y/N)?</B></TD>
<TD><B>RTF Cancelled (Y/N)?</B></TD>
<TD><B>Bank Pickup Date</B></TD>
<TD><B>Depositing Bank</B></TD>
<TD><B>Credit Date</B></TD>
<TD><B>Bounce Date</B></TD>
<TD><B>Reason of Bounce</B></TD>
<TD><B>Recovered Date</B></TD>
<TD><B>PA Minus Receipt Date</B></TD>
<TD><B>RTF Date Minus PA Date</B></TD>
<TD><B>Bank Pickup Date Minus RTF Date</B></TD>
<TD><B>Credit Date Minus Bank Pickup Date</B></TD>
<TD><B>Bounce Date Minus Credit Date</B></TD>
<TD><B>Transfer to Ingenium Date</B></TD>
<TD><B>Service Agent ID</B></TD>
<TD><B>Service Agent Name</B></TD>
<TD><B>Payment Reason</B></TD>
</TR>

<tr>
<%
if(htDataToExport != null && htDataToExport.size() > 0){
Hashtable htData = (Hashtable)htDataToExport.get("ExcelExport");
for(int i=0;i<htData.size();i++){
String[] arrRowData = (String[])htData.get("Row"+(i+1));
if(iFlag == 2){
%>
<tr>
<%
iFlag = 1;
}
else{%>
<tr style="background-color:grey">
<%
iFlag = 2;
}
for(int j=0;j<arrRowData.length;j++){
%>
<td> <%=JspUtil.checkNull(arrRowData[j])%> </td>
<%
}
%>
</tr>
<%
}
}
else{
%>
<tr> <td> No Data Available </td> </tr>
<%
}
}
catch(Exception ex){
out.println(ex.getMessage());
}
htDataToExport.clear();
GregorianCalendar gcBusinessDate = DateUtil.getBusinessDate();
%>
</table>

<table>
<TR></TR>
<TR></TR>
<tr>
<td ALIGN="RIGHT"><B>Generated on </B></TD>
<td></td>
<td ALIGN="RIGHT"><B>Date : </B></TD>
<td><%=DateUtil.retStrDate(gcBusinessDate)%></td>
<td ALIGN="RIGHT"><B>Time : </B></TD>
<td ALIGN="LEFT"><%=gcBusinessDate.HOUR%>:<%=gcBusinessDate.MINUTE%>:<%=gcBusinessDate.SECOND%></td>
<td ALIGN="RIGHT"><B>Verified By : </B></TD>
<td></td>
<td ALIGN="RIGHT"><B>Approved By : </B></TD>
<td></td>
</TR>
<TR></TR>
<TR><TD Colspan="9"><B>Information contained herein is confidential and for internal use and may be used only for business purposes authorized by xxxxxxxxxxxx</B></TD>
</TR>
</TABLE>

</form>
</BODY>
</HTML>
 
Tim Moores
Saloon Keeper
Posts: 7585
176
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't see any Excel export, I only see HTML being generated - do you want to rely on MS Excel being able to interpret the resulting HTML?
 
naga eswar
Ranch Hand
Posts: 138
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tim,

Sorry for late(very) reply. stuck up on other issue.

Below line of code will request the user's browser open the results with Excel

 
Tim Moores
Saloon Keeper
Posts: 7585
176
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
OK, so you are indeed not creating an Excel file, but want to use Excel to open an HTML file. For creating paginated HTML tables that have an export capability to Excel I recommend the DisplayTag tag library. (Note that, for that too, you'd have to rewrite your code so that the JSPs don't contain any Java code. But since you will have to do that anyway, you might as well start by doing that.)
 
naga eswar
Ranch Hand
Posts: 138
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Tim,

I am working on this Display Tag library. If any doubts i will get back.
 
Rancher
Posts: 2759
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don't think 15K records should lead to high memory consumption. It's not like you are doing millions of records. let's say each column has 50 characters each, and you have 40 columns, so 15K*40*50 = 30M, which is not too much The approach that you have is not scalable, in the sense that, if you suddenly jump from 15K records to 15M it will stop working. However, it should work for 15K records. There might be something else going on that might be using memory.

We have several places where we export data like this, and the approach that works is to export the data as a CSV in a servlet without involving a JSP. You can set the content type to text/csv and Excel will be able to open and display it. Inside the for loop that iterates over the resultset, format the row as a CSV and directly write it to the servlet's output stream. This way you don't have to keep any rows in memory, they get sent to the client directly.

The only memory usage will be
a) Most JDBC drivers prefetch rows from database. If you ask row 1, it will fetch row1-50 and buffer it. The batch size is configurable. You will have that many rows in memory
b) ServletOutputStream buffers the output(I think). The buffer will use that memory

Both these buffers should be small and constrained. You shouldn;t run out of memory because of these buffers.

If you really want to export a true Excel file, you will have to figure out how to stream output to an Excel file. I am not sure if it's possible. I think POI has a streaming API. I don't know if it supports streaming output or works only on input
 
reply
    Bookmark Topic Watch Topic
  • New Topic