The problem was that once Rampart is engaged, it expects the response to have the same security header. The way I solved the problem was by removing the handler to the Inflow security in the Rampart.mar file.
I'm not sure if this is the best fix, but it worked for us.
To remove the inflow handler: Unpack the rampart.mar file
Comment out the Inflow section
Zip up the META_INF folder. Then rename the .zip file to be .mar
Now when you use this as there are no handlers defined for inflow, it will just use the standard Axis2 response handler.
I guess if you had several projects using Rampart where some had the security header in the response and some didn't you would need a different approach.