Not much I can say, I'm afraid, without seeing javacore* and heapdump* files.
What WebSPhere version are you running exactly (run the wasversion script to get that data) The first and most obvious step, then, is to make sure you're on the latest fixpack.
Also, there is often useful informatoin in the FFDC logs. I suggest you sort them by date modified to find the useful ones.
I also suggest you turn on verbose GC to assure that you are not just legitimately running out of heap space.
It APPEARS that this is happening on a new TCP/IP Socket connection (to be clear, this is a guess), so I would concur with conclusions I've seen you make in another forum that this may well be an app server problem. Are you running Stand-alone WAS and running against the web container within the app server?