This week's book giveaway is in the OO, Patterns, UML and Refactoring forum. We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line! See this thread for details.
Trail and error mostly. Some apps hog memory and others dont. It depends on how state is maintained; what is cached; how large the cache is; what the purge policy for caches are; etc etc.
A load and performance test usually highlights the limits of CPU / memory / Threads. A good rule of thumb is to add some buffer to the max limit (say 20%. But 20% of what number ? That can be a factor too).