Use JAVA_HOME to specify which Java JDK you want to compile with.
Setting the "source" attribute merely tells the compiler what source restrictions apply. For example, setting 'source="1.3"' when using JDK 1.5's compiler should allow code with variables named "enum" to compile (enum is a reserved word in Java 1.5).
The "target" attribute likewise indicates what java level, the generated class files should be built for. One reason you'd set this is if you're building classes that will be serialized, since serialization isn't guaranteed across multiple runtime JVM versions. A major pain if you're doing RPC-style stuff.
Sometimes the only way things ever got fixed is because people became uncomfortable.