In general, catalina home should refer to the directory where tomcat is installed. For NetBeans 5.0, it would be something similar to:
<ide_install_dir>/nb50/ide/enterprise2/jakarta-tomcat-5.5.9
As per
http://tomcat.apache.org/tomcat-5.0-doc/appdev/deployment.html: The description below uses the variable name $CATALINA_HOME to refer to the directory into which you have installed Tomcat 5, and is the base directory against which most relative paths are resolved. However, if you have configured Tomcat 5 for multiple instances by setting a CATALINA_BASE directory, you should use $CATALINA_BASE instead of $CATALINA_HOME for each of these references.
In the IDE, can you try selecting the radio option that lets you set the base directory?