That probably (and it's just a guess, I've never used FOP for generating RTF) reflects the fact that PDF is by far the most widely used output format of FOP, and RTF -and possibly some of the other formats- linger somewhere between an afterthought and a neglected stepchild. From the FOP web site: "At the moment FOP is mainly a tool to render XSL-FO files to pdf.", and looking at the open issues for FOP's RTF functionality -some of which are several years old- I wouldn't get my hopes up that FOP's RTF support improves any time soon. However, anecdotal reports talk of "... render 7000 page RTF and PDF reports that contain continuous tables of over 20,000 records seamlessly (without page breaks in the table that are not at an end of page) without OOM errors.", so YMMV.
I’ve looked at a lot of different solutions, and in my humble opinion Aspose is the way to go. Here’s the link: http://aspose.com