FOP - Out of memory

[Jeppe Sommer]

Hi.

I am trying to use FOP to get an e-mail attachment returned as byte array.

It seems that I am running out of memory when I create the Fop object:
Fop fop = fopFactory.newFop(MimeConstants.MIME_PDF, foUserAgent, out);

I am pretty sure it has something to do with the ByteArrayOutputStream.

Any help would be appreciated?

public byte [] getTicketAttachment( String ticketsAsString, String foURL )throws Exception {
	     
		 js.util.Date obj_date = new  js.util.Date();
	     byte [] attachment = null;
		 ByteArrayOutputStream out = null;
		 
		 try {
		 
		 	   FopFactory fopFactory = FopFactory.newInstance();
    	 	   TransformerFactory tFactory = TransformerFactory.newInstance();
	
		 	   out = new ByteArrayOutputStream();
			   
		 	   FOUserAgent foUserAgent = fopFactory.newFOUserAgent();
		  
		  	   System.out.println("***********************test****: "+obj_date.getTimestampAsString()); 
		 	   //Setup FOP
		 	   Fop fop = fopFactory.newFop(MimeConstants.MIME_PDF, foUserAgent, out);
		 	   System.out.println("***********************Here it runs out of memory****: "+obj_date.getTimestampAsString());   
		 	   //Setup Transformer
		 	   // Setup XSLT
         	   Transformer transformer = tFactory.newTransformer(new StreamSource(foURL));
		 	   System.out.println("***********************test 1****: "+obj_date.getTimestampAsString());
		 	   // Set the value of a <param> in the stylesheet
         	   transformer.setParameter("versionParam", "2.0");
		 	   Result result = new SAXResult(fop.getDefaultHandler());
		 	   System.out.println("***********************test 2****: "+obj_date.getTimestampAsString());
		 	   Source src = new StreamSource( new StringReader( ticketsAsString ) );
		 	   System.out.println("***********************test 3****: "+obj_date.getTimestampAsString());
		 	   transformer.transform( src, result );
		 	   System.out.println("***********************test 4****: "+obj_date.getTimestampAsString());
 		 	   attachment = out.toByteArray();
		 }
		 finally {
		       
			   if ( out != null )
			   out.close();
		 }
		 return attachment;
		 
	 } 

Server log:

***********************test****: 20130730143214
javax.servlet.ServletException: Servlet execution threw an exception
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:313)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at js.util.RedirectFilter2.doFilter(RedirectFilter2.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:662)
Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
Jul 30, 2013 2:33:43 PM org.apache.catalina.startup.ClassLoaderFactory validateFile

[Ulf Dittmer]

How big is the attachment? How much memory does the servlet container have allocated? FOP needs a lot.

[Jeppe Sommer]

The file is only 12 kb.

The server have 2560 MB and Xmx512m.

I have tried to save the attachment as a file on disk.

The error now come here:
transformer.transform( src, result );

public byte [] getTicketAttachmentFromFile( java.io.File ticket, String foURL )throws Exception {
	     
		 js.util.Date obj_date = new  js.util.Date();
	     byte [] attachment = null;
		 ByteArrayOutputStream out = null;
		 
		 try {
		 
		 	   FopFactory fopFactory = FopFactory.newInstance();
    	 	   TransformerFactory tFactory = TransformerFactory.newInstance();
	
		 	   out = new ByteArrayOutputStream();
			   
		 	   FOUserAgent foUserAgent = fopFactory.newFOUserAgent();
		  
		  	   //Setup FOP
		 	   Fop fop = fopFactory.newFop(MimeConstants.MIME_PDF, foUserAgent, out);
		 	   //Setup Transformer
		 	   // Setup XSLT
         	   Transformer transformer = tFactory.newTransformer(new StreamSource(foURL));
		 	   System.out.println("***********************test 1 time****: "+obj_date.getTimestampAsString());
		 	   // Set the value of a <param> in the stylesheet
         	   transformer.setParameter("versionParam", "2.0");
		 	   Result result = new SAXResult(fop.getDefaultHandler());
		 	   System.out.println("***********************test 2 time****: "+obj_date.getTimestampAsString());
		 	   Source src = new StreamSource( ticket );
		 	   System.out.println("***********************test 3 time****: "+obj_date.getTimestampAsString());
		 	   transformer.transform( src, result );
		 	   System.out.println("***********************test 4 time****: "+obj_date.getTimestampAsString());
 		 	   attachment = out.toByteArray();
		 }
		 finally {
		       
			   if ( out != null )
			   out.close();
		 }
		 return attachment;
		 
	 }

Server log:

***********************test 1 time****: 20130731021907
***********************test 2 time****: 20130731021907
***********************test 3 time****: 20130731021907
Jul 31, 2013 2:19:16 AM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
	at java.lang.Throwable.getStackTraceElement(Native Method)
	at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
	at java.lang.Throwable.printStackTrace(Throwable.java:510)
	at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
	at org.apache.juli.FileHandler.publish(FileHandler.java:198)
	at java.util.logging.Logger.log(Logger.java:481)
	at java.util.logging.Logger.doLog(Logger.java:503)
	at java.util.logging.Logger.logp(Logger.java:703)
	at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
	at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:274)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:662)
Exception in thread "http-80-1" java.lang.OutOfMemoryError: PermGen Space

[Tim Holloway]

Oh yuck. PermGen space.

PermGen memory isn't the same thing as regular heap memory. For Tomcat, the default PermGen space is "only" 128MB, but if you're using an ORM or other systems that do a lot of static objects, it's often advisable to double that.

If you are running Tomcat, search the Tomcat forum for more on this particular issue.

[Richard Tookey]

I find it worrying that this 'out of memory' is happening inside a Servlet. If the Servlet handles many simultaneous requests then it is possible that two or more FOP transformation will be active at the same time and if you are short of memory with just one request then, obviously, you are going to be worse off with two or more simultaneous requests. You may have to consider having a process, maybe external to the Servlet container, which queues FOP transformations to make sure that you limit the number of active transformations. Of course this could mean that your average response time will increase but that must be better than an 'out of memory' condition.

[Tim Holloway]

Richard Tookey wrote:I find it worrying that this 'out of memory' is happening inside a Servlet. If the Servlet handles many simultaneous requests then it is possible that two or more FOP transformation will be active at the same time and if you are short of memory with just one request then, obviously, you are going to be worse off with two or more simultaneous requests. You may have to consider having a process, maybe external to the Servlet container, which queues FOP transformations to make sure that you limit the number of active transformations. Of course this could mean that your average response time will increase but that must be better than an 'out of memory' condition.

Depending on the size and complexity of the request, running FOP inside a servlet is not an unrealistic thing to do. If you want a web request to generate a PDF via FOP, ultimately, it's going to be a servlet receiving the request anyway. The apache Cocoon webapp framework relies on FOP for its PDF functionality, for example.

The issue here isn't "running out of memory", it's running out of PermGen memory. PermGen space isn't the same thing as general space. It's allocated differently, and, as I mentioned before, it's not as big as the regular heap memory pool. Nor can adjusting the heap limits help, since there's a completely separate parameter that sets the PermGen size.

[Richard Tookey]

Tim Holloway wrote:
The issue here isn't "running out of memory", it's running out of PermGen memory. PermGen space isn't the same thing as general space. It's allocated differently, and, as I mentioned before, it's not as big as the regular heap memory pool. Nor can adjusting the heap limits help, since there's a completely separate parameter that sets the PermGen size.

OK it is not running out of general memory but 'PermSpace' but surely this doesn't change the fact that FOP is consuming a too much of the container's resources for just one request and simultaneous requests will make it far worse. Many year ago when I first used FOP in earnest the FOP transformation took so much memory (JDK1.4 so not PermGen involved) that the FOP processing was done in a separate computer.

[Tim Holloway]

Actually, we have no idea how much general heap memory this particular transformation is using. All we truly know is that it wants more PermGen space. And lacking actual breakdowns, I can't even guarantee that FOP is what's eating PermGen. I've never had FOP blow out that particular resource, although ORM-based apps are a perennial nuisance in that regard for me. So FOP+Hibernate could be more a Hibernate problem than a FOP problem, as an example.

I would definitely monitor system resource usage, however. There may be FOP-related issues. Although a "lot" of memory is something whose absolute size changes from year to year. Still, in order to measure how much heap memory is being eaten - and whether it's worth the expense to redesign for an external processor - first we need to resolve the immediate problem. That can be done with a simple VM setting, which is a lot faster and easier than re-architecting the app for gains which may or may not materialize.

[Richard Tookey]

Tim Holloway wrote:, which is a lot faster and easier than re-architecting the app for gains which may or may not materialize.

I have voiced my concerns about FOP memory resource usage and have not suggested a "re-architecting". Having worked on a project that employed an IBM high volume web site consultant I know that establishing a client usage pattern is paramount in deciding on an architecture. Knowing the usage pattern allows one to model various architectures to establish resources required. My experience is that FOP is (or at least was) a very resource hungry process and may be bad news when used in a web site. I know little about the more recent garbage collectors and bow to your experience but I wanted the OP to be aware of possible resource shortage problems. I don't want to get into an argument over this.

[Jeppe Sommer]

Thanks for your comments.

We solved our memory issue by simply add:

-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:MaxPermSize=256m

For some reasons that we don't know, the MaxPermSize was reseted and too low.

When this is said our traffic is still low so I am not sure about your discussion.

I will keep in mind about the simultaneous active transformations...

[Tim Holloway]

MaxPermSize defaults to 128m. 256m will almost always do if you need more, although I did have to do a bump up to 384m on a complex system this morning.

Fortunately, PermGen space is not used for most memory needs and it's pretty much all shared memory, so adding more users is unlikely to require massive amounts of PermGen.