You description reminds me of a problem I had experienced problems on 32-bit Windows, where it was not possible to allocate more than about 1 GB of memory to the JVM, although a lot more of memory was available. The reason was that the JVM requires continuous virtual address space in memory, which was not available due to some device drivers and/or DLLs fragmenting the virtual address space (the DLLs must be mapped to the same virtual address in all processes, therefore loading a DLL at certain virtual memory address range makes that address range occupied in all future processes on the computer). Note that all this business bears no direct relation to the actual memory allocation, just reservation of the virtual address space. (Virtual address space fragmentation is quite complicated problem and I don't recall all the details from the top of my head, but I believe this is the gist of it.)
This problem doesn't exist in 64-bit Windows, because the virtual memory space is so big (2^64) that it is always possible to find many GBs of continuous virtual address space there. I'm not sure whether similar problems exist in other 32-bit systems (Unix, Mac-OS).