This week's book giveaway is in the OO, Patterns, UML and Refactoring forum. We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line! See this thread for details.
Try to run the transformation outside of WSAD with just the transformation-related code (you can use this program), give the JVM some 128MB of heap and see if that's enough. Then, start to cut down the maximum heap size until you start getting the OutOfMemoryError.
You might also try to debug the code by "stepping into" the XSLT engine and see what it's trying to do when it runs out of memory. Also, a bit easier way could be to run a profiler against the code doing a transformation -- it might tell you where all that memory is going.
I’ve looked at a lot of different solutions, and in my humble opinion Aspose is the way to go. Here’s the link: http://aspose.com