Well...seems like you have an improper installation of Java.
1. Remove Java installation from your machine, install it afresh and try again.
2. Write a simple hello world program and try compiling it. So that you will get more meaningful error message. It is a possibility that some variable is not properly set or something like that. Trace it down and remove.
For Tomcat, have a look at the script that you use to start up Tomcat. I don't know exactly what it's called, probably tomcat.bat or catalina.bat in the bin directory of your Tomcat installation directory. You could add the -Xmx setting to the line in that script that starts Tomcat.