Assume you are running from the command line, Maven is run via the mvn (or mvn.bat) script in m2_home/bin. You edit that file, modifying the Java run command. Or you can add a MAVEN_OPTS=xxx line early in that file. Or you could even set the MAVEN_OPTS env var before running maven.
Also, set _JAVA_LOADER_DEBUG (you can set it to anything). This will cause the JVM to dump some startup information, including the JVM options it was given. This should help determine if the heap size is being set.