The largest possible heap size depends on the largest chunk of consecutive memory that the JVM can allocate (not all JVM need consecutive memory, but Sun's does). If you are running a 32-bit JVM on a 32-bit OS, then you need to understand how memory is laid out. Each app has only 4GB of addressable space. Typically, the OS uses 2GB and lets the app have the other 2GB. The app's address space it used up by usually by the exe in low memory and the OS DLLs in high memory just under the 2GB border (even the JVM DLLs load up into high memory). This leaves about a 1.5GB contiguous area in memory where the JVM can allocate the heap. The actual size grows and shrinks depending on the OS version and patches.
There is a /3GB os boot option you can use to split the app's memory such that the OS uses only 1GB and the app gets 3GB. Unfortunately, the OS DLLs (and JVM DLLs) still occupy the area just under 2GB, so you still end up with only 1.5GB of contiguous space. If you use a JVM that does not need contiguous memory to allocate the heap you can get about 2.7GB heaps. Of course, the /3GB options does have its down sides. Having only 1GB for its memory structures (threads, file handles, object handles, etc) can cause the OS to run out of resources. Usually you see this when the JVM is unable to create another thread. And app server create lots of threads.
What operating system are you using? I assume that it is a 32-bit OS. What is the memory usage and availability at the time you launch Tomcat and JBoss AS? What is the full command line used to launch Tomcat and JBoss AS? What is the largest acceptable max heap size for JBoss AS? There are dozens of things that can affect how big a heap you can specify, so I will probably have you running lots of things and looking at various things to pinpoint this down.