This week's book giveaway is in the OCAJP 8 forum. We're giving away four copies of OCA Java SE 8 Programmer I Study Guide and have Edward Finegan & Robert Liguori on-line! See this thread for details.
Try to run the transformation outside of WSAD with just the transformation-related code (you can use this program), give the JVM some 128MB of heap and see if that's enough. Then, start to cut down the maximum heap size until you start getting the OutOfMemoryError.
You might also try to debug the code by "stepping into" the XSLT engine and see what it's trying to do when it runs out of memory. Also, a bit easier way could be to run a profiler against the code doing a transformation -- it might tell you where all that memory is going.