WAS server debug mode is sometimes very unstable. There's no simple fix since your description was pretty vague but you might want to try installing any IBM fix packs for your particular version of both WSAD/WAS.
Also, try starting your server in debug mode outside of WSAD (using WSADMIN) then connecting with WSAD using a Remote Java Application call supported in the WSAD debug menu.
Hi, I got exactly same issue when I run the application in debug mode.
In WAS admin console >servers >application server. Then click on Server1 > Process Defintition>Java Virtual Machine. Here you see all general properties and increase Initial Heap Size =256 and Maximum Heap Size = 512. THis fix the above problem.