This week's giveaways are in the MongoDB and Jobs Discussion forums. We're giving away four copies of Mongo DB Applied Patterns and 4 resume reviews from Five Year Itch and have the authors/reps on-line! See this thread and this one for details.
Trail and error mostly. Some apps hog memory and others dont. It depends on how state is maintained; what is cached; how large the cache is; what the purge policy for caches are; etc etc.
A load and performance test usually highlights the limits of CPU / memory / Threads. A good rule of thumb is to add some buffer to the max limit (say 20%. But 20% of what number ? That can be a factor too).