First of all I have to say that I do not have an immediate solution for your problem.
I am wondering however why you are not using the Java API for VLC (
JVLC). Maybe this will help you to have some more control over VLC.
But if you really want to do it like you are doing at the moment, some questions come to mind. Have you tried running the .sh script itself (without java program)? Does it work correctly?
Have you tried using the runtime without the .sh script? You can also execute the vlc command directly, maybe that solves the problem.
Sorry that I cannot be more helpful.