What threshold of memory consumtion can make a java process resulted in non responsive mode?
This could be tricky, as obviously something wrong either a memory leak or insufficient memory, but still is there a way to decide a memory consumption threshold so that a service or process can be killed manually before a level of non-responsive mode due to memory usage increasing gradually.
In my experience, Java programs becoming non responsive has nothing to do with memory consumption and everything to do with Threads getting stuck in loops or waiting for unavailable resources.
Using large amounts of memory might result in the operating system thrashing as it pages memory to disk, but that is slow, not unresponsive and depends on the operating system, available real memory, other processes, etc.
Actually running out of memory will of course throw an error.