Trail and error mostly. Some apps hog memory and others dont. It depends on how state is maintained; what is cached; how large the cache is; what the purge policy for caches are; etc etc.
A load and performance test usually highlights the limits of CPU / memory / Threads. A good rule of thumb is to add some buffer to the max limit (say 20%. But 20% of what number ? That can be a factor too).