Multi-Thread command line executor cause uncertain result

[Zee Ho]

All, I have writen a Java program to execute the external program.
sth like execute "cat < abc.xml" and such java program will return same result as linux command.


I test the program use such way : give my program 64 same file and
let it execute them. In some of the machine the task 100% completed, while in some machine several task failed with exceptions as :

- 2005-10-12 [Thread-712] java.io.IOException: Stream closed
OR
- 2005-10-12 [Thread-878] java.io.IOException: Bad file descriptor
(I am still the file is not missing)

I know using thread is error-prone, but I can not figure out what mistake I have made. any suggestion on ehancing the program?

package com.core.cif.util;
 
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
 
import org.apache.commons.io.CopyUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang.SystemUtils;
import org.apache.log4j.Logger;
 
import com.core.esg.SystemException;
/**
 * To execute external command line program.
 */
public class CommandLineExecuter {
    private static final Logger LOGGER =
        Logger.getLogger(CommandLineExecuter.class);
    /** The executable command */
    protected String command;
    /**Stands for the type of stream*/
    public static final String TYPE_OUTPUT = "output";
    /**Stands for the type of stream*/
    public static final String TYPE_ERROR = "error";
    /**
     * Instantiate the class.
     * @param command the external command to execute
     */
    public CommandLineExecuter(final String command) {
        this.command = command;
    }
    /**
     * Execute the external program without any parameter and
     * return the status code which indicates the program normally
     * terminated or not
     * @return the status code
     * @throws SystemException if any exception occurs
     */
    public int execute() throws SystemException {
       return execute(null);
    }
    /**
     * Execute the external program with enviroment arguments and
     * return the status code which indicates the program normally
     * terminated or not
     * @param args the enviroment arguments
     * @return the status code
     * @throws SystemException if any exception occurs
     */
    public int execute(final String[] args) throws SystemException {
        return execute(args, null);
    }
    /**
     * Execute the external program with enviroment arguments and
     * ebxml bytes, the method will return the status code which
     * indicates the program normally terminated or not
     * @param args the enviroment arguments
     * @param input the input bytes which indicates the ebxml.
     * @return the status code
     * @throws SystemException if any exception occurs
     */
    public int execute(final String[] args, final byte[] input)
            throws SystemException {
        return ((Integer)
                    this.executeExternalPrgm(args, input, false)).intValue();
    }
    /**
     * Execute the external program without parameters,
     * the method will return the output of the external
     * program as the byte array.
     * @return the output of the external program.
     * @throws SystemException if any exception occurs
     */
    public byte[] executeAndOutputAsBytes() throws SystemException {
        return executeAndOutputAsBytes(null);
    }
    /**
     * Execute the external program with enviroment arguments
     * , the method will return the output of the external
     * program as the byte array.
     * @param args the enviroment arguments
     * @return the output of the external program.
     * @throws SystemException if any exception occurs
     */
    public byte[] executeAndOutputAsBytes(final String[] args)
        throws SystemException  {
        return executeAndOutputAsBytes(args, null);
    }
    /**
     * Execute the external program with enviroment arguments and
     * ebxml bytes, the method will return the output of the external
     * program as the byte array.
     * @param args the enviroment arguments
     * @param input the input bytes which indicates the ebxml.
     * @return the output of the external program.
     * @throws SystemException if any exception occurs
     */
    public byte[] executeAndOutputAsBytes(
            final String[] args, final byte[] input)
            throws SystemException {
        return ((byte[]) executeExternalPrgm(args, input, true));
    }
    /**
     * Execute the command which specified when creating
     * object with the parameter and values passed in.
     * @param args
     * @param input
     * @param needOutput
     * @return the status code which stands that the program
     * is normally terminated or not.
     * @exception SystemException if any error occurs
     */
    private Object executeExternalPrgm(
            final String[] args, final byte[] input,
            final boolean needOutput)
            throws SystemException {
        InputStream stderr = null;
        InputStream inputStream  = null;
        InputStream procInputStream = null;
        OutputStream os  = null;
        ByteArrayOutputStream inHandlerOut = new ByteArrayOutputStream();
        ByteArrayOutputStream errHandlerOut = new ByteArrayOutputStream();
        Object result = null;
        try {
            Runtime rt = Runtime.getRuntime();
            Process proc = rt.exec(getCommand(), args);
            //proc.getOutputStream() is the input argument of the command.
            os = new BufferedOutputStream(proc.getOutputStream(), 1024);
            //copy the argument input to the output stream.
            //eg. cat < abc.xml
            //then the method argument input stands for the content of abc.xml
            //and it is necessary to provide a output stream for cat command
            //then we will use the IOHandler to copy the abc.xml content
            //to the output stream.
            if (input != null) {
                inputStream = new ByteArrayInputStream(input);
                CommandLineIOHandler outHandler =
                    new CommandLineIOHandler(TYPE_OUTPUT,
                            inputStream, os);
                outHandler.start();
                outHandler.join();
                IOUtils.closeQuietly(inputStream);
                IOUtils.closeQuietly(os);
            }
            procInputStream = proc.getInputStream();
            stderr = proc.getErrorStream();
            CommandLineIOHandler inHandler =
                    new CommandLineIOHandler(TYPE_OUTPUT,
                            procInputStream, inHandlerOut);
            CommandLineIOHandler errHandler =
                    new CommandLineIOHandler(TYPE_ERROR, stderr, errHandlerOut);
            errHandler.start();
            inHandler.start();
            int status = proc.waitFor();
            if (needOutput) {
                result = inHandlerOut.toByteArray();
                LOGGER.debug("Result is : "
                        + new String((byte[]) result));
            } else {
                result = new Integer(status);
                LOGGER.debug("Result is : " + result);
            }
        } catch (IOException e) {
            throw new SystemException(
                "Failed to execute the command.1 : " + command , e);
        } catch (InterruptedException e) {
            throw new SystemException(
                "Failed to execute the command.2 : " + command , e);
        } finally {
            IOUtils.closeQuietly(stderr);
            IOUtils.closeQuietly(procInputStream);
            IOUtils.closeQuietly(inHandlerOut);
            IOUtils.closeQuietly(errHandlerOut);
        }
        return result;
    }
    /**
     * Gets the OS specific command
     * @return the command.
     */
    protected String[] getCommand() {
        String[] command = null;
        if (SystemUtils.IS_OS_WINDOWS) {
            if (SystemUtils.IS_OS_WINDOWS_95) {
                command = new String[]{"command.com", "/C", this.command};
            } else {
                command = new String[]{"cmd.exe", "/C", this.command};
            }
        } else {
           command = new String[]{this.command};
        }
        return command;
    }
    /**
     * The handler of the CommandLineExecuter.
     */
    class CommandLineIOHandler extends Thread {
        private String type;
        private InputStream is;
        private OutputStream os;
        /**
         * Instantiate the class.
         * @param type the type of the output
         * @param is the input stream.
         */
        public CommandLineIOHandler(final String type,
                final InputStream is) {
            this(type, is, null);
        }
        /**
         * Instantiate the class.
         * @param type the type of the output
         * @param is the input stream.
         * @param os the output stream
         */
        public CommandLineIOHandler(final String type,
                final InputStream is, final OutputStream os) {
            this.type = type;
            this.is = is;
            this.os = os;
        }
        /**
         * @see Thread#run()
         */
        public void run()  {
            try {
                if (this.os != null) {
                    CopyUtils.copy(is, os);
                } else {
                    LOGGER.info(this.type + " : " + IOUtils.toString(is));
                }
            } catch (IOException e) {
                LOGGER.debug("Failed to process external command", e);
            }
        }
    }
}
 

[Ernest Friedman-Hill]

Please CarefullyChooseOneForum for your questions. Don't post the same question to multiple forums. I've deleted the one in Java in General (Beginner), as this is not what I'd call "beginner" code.

So what's different about the machines where this fails? What OS are they running? Are they all the same?

[Zee Ho]

No one is suse unix and another is redhat linux.

in redhat, seems 100% complete.

[Ernest Friedman-Hill]

Are you running the Sun JVM on both, or is the failing one Kaffe or some other open-source VM?

[Zee Ho]

I run on same Sun's java vm

As I can not find out a way to make it stable I changed my way to execute the external program

that I develop a .sh file which take 3 parameter as $1 < $2 > $3

then the way I run the arbitrary program become
1. obtain the program path for $1
2. write the input file to file system rather than manipulating the input/output stream through the Process class. $2 became the input file path
3. offer the destination file path for $3

then let the underlying OS execute the .sh became more stable than my previous solution

would like share this idea to ranchers if helpful
[ October 13, 2005: Message edited by: Zee Ho ]