aspose file tools*
The moose likes Java in General and the fly likes How to accomplish this with one JVM instance Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Java » Java in General
Bookmark "How to accomplish this with one JVM instance" Watch "How to accomplish this with one JVM instance" New topic
Author

How to accomplish this with one JVM instance

Ricky Murphy
Ranch Hand

Joined: Oct 16, 2007
Posts: 31
Hello:

Trying to get some suggestions to resolve this issue: I have a set of 300 unix shell scripts (jobs) , each will pass in a set of parameters to a java program (the same java program) for further processing.

The first solution came to me was that I wrote a MyApp class with a main() method, then each of these unix shell script will call: "java MyApp param1, param2..." the problem of this is that it will launch too many JVM instances.

I am wondering is there a way that I only launch and use one JVM instance and all 300 calls become 300 threads? I thought about making a web app and deploy it to Tomcat. However, is there a way to avoid making it a web app?

Please shed some light.

Thank you.

RickM
Martin Vajsar
Sheriff

Joined: Aug 22, 2010
Posts: 3610
    
  60

First of all, you don't want to have 300 threads running in parallel. You need to create only so many threads your HW can handle. If your threads start to compete for CPU, or - worse - IO, the overall performance will degrade, perhaps massively.

Alleviating the overhead of starting the JVM 300 times might be a good idea, especially if the jobs are relatively fast. If it is not the case, eg. one job runs for a minute or more on average, I'd say the JVM creation overhead could be acceptable.

If you do want to process everything in one go, then I'd suggest storing the parameters for all 300 jobs into some sort of a text file (easily doable by a shell script on Unix) - one line per job. Your app would then read the job parameters from it and could easily distribute them among worker threads it would manage. A producer/consumer design pattern looks like a good match for your needs - try to google it up if you didn't hear about it yet.

Edit: from your description I understood your jobs are some sort of utility jobs, with no user interaction. If that is the case I don't see any benefit in turning it into a web app.
Ricky Murphy
Ranch Hand

Joined: Oct 16, 2007
Posts: 31
Thank you Martin for the valuable analysis and suggestions. Let me provide more detailed background:

The java app are indeed utility kind, no user interface. Basically, each of those unix scripts calls the java app with a source and a target directory name. The java program searches the source directory and calls another web service to upload the files found in the source folder into the target directory in a document repository. The unix scripts, my java app, and the web service are located on the same unix box. The web service and the document repository are provided by a vendor. I only need to provide the java program to connect the unix shell scripts and the web service.

As you can see, depends on how many files found in the source directory, each call could last to 1 or 2 min, if not longer. Hence, if I start the jobs using: java MyApp arg1, arg2... chances are 300 jvm instances could run at the same time. I am not sure how big the overhead of having 300 JVM instances running at the same time, could you please comment on it? Or, what if I turn my java app into a web service (basically it means to wrap the vender service calls inside my service), 300 requests will therefore be handled by the app server, that means the app server will manage the 300 threads. I was originally trying to avoid another web app (i.e. web service), but let me bring it up for the discussion, it maybe the feasible approach.

As to your suggestion of have one job taking params from a text file, it wont fit the bill for my situation, because our jobs were created and configured with with different dependencies, In/Out conditions. etc (~300 is just the last step, as there are many more steps ahead it).

Thank you again,


RickM
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12805
    
    5
As to your suggestion of have one job taking params from a text file, it wont fit the bill for my situation, because our jobs were created and configured with with different dependencies, In/Out conditions. etc (~300 is just the last step, as there are many more steps ahead it).


So create the job specification with a more flexible format - XML comes to mind. A Java object using the Command Pattern approach would also work.

Thats what I would do if this was my problem, and I would then experiment with the number of Threads in a Thread pool to which these jobs could be assigned, monitoring memory and CPU use.

Of course, if multiple computers were available then you could farm out the jobs with something like JavaSpaces, but for a single CPU there is no reason to have more than one JVM with multiple worker threads.

Bill
Ricky Murphy
Ranch Hand

Joined: Oct 16, 2007
Posts: 31
Thanks guys. It makes a lot of sense. I will take it forward and hope that I can break the mold...
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: How to accomplish this with one JVM instance