Well...seems like you have an improper installation of Java.
1. Remove Java installation from your machine, install it afresh and try again.
2. Write a simple hello world program and try compiling it. So that you will get more meaningful error message. It is a possibility that some variable is not properly set or something like that. Trace it down and remove.
When you run a Java program, you can use the -Xmx switch on the command line to specify how much memory Java can allocate at most. For example, to let Java use up to 256 MB memory, run it like this:
java -Xmx256m com.mypackage.MyProgram
For Tomcat, have a look at the script that you use to start up Tomcat. I don't know exactly what it's called, probably tomcat.bat or catalina.bat in the bin directory of your Tomcat installation directory. You could add the -Xmx setting to the line in that script that starts Tomcat.