The message indicates that you have set some heap size which is higher than what your machine can make available. What is your machine configurations and what heap size have you set while starting JBoss?
You get error since it is using up too much memory while starting up. You can control the memory that it used in the file run.conf under the bin directory of jboss. Open run.conf and search for 'mem' and you will see the line where you can set the memory which will used for the heap. It was 256 and 512 for mine and i changed to 128 and 256 and it worked fine.
I hope this is a typo because a heap setting of 128 bytes min and 256 bytes max is not going to work. Yes, bytes, not megabytes or kilobytes, bytes.
It would be better if you posted the exact settings you made (the JAVA_OPTS settings from the startup script), and posted the exact error you got. And you should have started a new post rather than reviving one that started 6 years ago - creating a new post would have forced you to provide details. See ItDoesntWorkIsUseless