I would proceed in this order.
1&2. Make sure that connections from the connection pool are properly closed and disposed of. Your observation that it takes about 100 queries to hit the error suggests that connections are not being properly recycled. Also be sure that all
JDBC objects are properly closed/released/whatever.
3. Provide for logging (or just System.out.print) memory use after each request. See the java.lang.Runtime methods.
The number of Threads available in Tomcat to process requests is set in server.xml. The configuration of the connection pool probably is done with some sort of initialization parameters - see the rpDBPool docs. Of course it makes no sense to have more connections than request threads.
There is nothing to be gained by multiple instances of Tomcat on the same machine, if you give Tomcat enough memory. If you have a multiprocessor machine, it will use both automatically.
As I recall, you can set the command line parameters such as memory as the "JAVA_OPTS" environment value or in the catalina.bat file.
The documentation that comes with your tomcat installation is the most coherent online docs I have found. There is a new edition of "Professional Apache Tomcat" - I found the previous edition to be very useful.
Bill