• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

How to accomplish this with one JVM instance

 
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello:

Trying to get some suggestions to resolve this issue: I have a set of 300 unix shell scripts (jobs) , each will pass in a set of parameters to a java program (the same java program) for further processing.

The first solution came to me was that I wrote a MyApp class with a main() method, then each of these unix shell script will call: "java MyApp param1, param2..." the problem of this is that it will launch too many JVM instances.

I am wondering is there a way that I only launch and use one JVM instance and all 300 calls become 300 threads? I thought about making a web app and deploy it to Tomcat. However, is there a way to avoid making it a web app?

Please shed some light.

Thank you.

RickM
 
Sheriff
Posts: 3837
66
Netbeans IDE Oracle Firefox Browser
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
First of all, you don't want to have 300 threads running in parallel. You need to create only so many threads your HW can handle. If your threads start to compete for CPU, or - worse - IO, the overall performance will degrade, perhaps massively.

Alleviating the overhead of starting the JVM 300 times might be a good idea, especially if the jobs are relatively fast. If it is not the case, eg. one job runs for a minute or more on average, I'd say the JVM creation overhead could be acceptable.

If you do want to process everything in one go, then I'd suggest storing the parameters for all 300 jobs into some sort of a text file (easily doable by a shell script on Unix) - one line per job. Your app would then read the job parameters from it and could easily distribute them among worker threads it would manage. A producer/consumer design pattern looks like a good match for your needs - try to google it up if you didn't hear about it yet.

Edit: from your description I understood your jobs are some sort of utility jobs, with no user interaction. If that is the case I don't see any benefit in turning it into a web app.
 
Ricky Murphy
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you Martin for the valuable analysis and suggestions. Let me provide more detailed background:

The java app are indeed utility kind, no user interface. Basically, each of those unix scripts calls the java app with a source and a target directory name. The java program searches the source directory and calls another web service to upload the files found in the source folder into the target directory in a document repository. The unix scripts, my java app, and the web service are located on the same unix box. The web service and the document repository are provided by a vendor. I only need to provide the java program to connect the unix shell scripts and the web service.

As you can see, depends on how many files found in the source directory, each call could last to 1 or 2 min, if not longer. Hence, if I start the jobs using: java MyApp arg1, arg2... chances are 300 jvm instances could run at the same time. I am not sure how big the overhead of having 300 JVM instances running at the same time, could you please comment on it? Or, what if I turn my java app into a web service (basically it means to wrap the vender service calls inside my service), 300 requests will therefore be handled by the app server, that means the app server will manage the 300 threads. I was originally trying to avoid another web app (i.e. web service), but let me bring it up for the discussion, it maybe the feasible approach.

As to your suggestion of have one job taking params from a text file, it wont fit the bill for my situation, because our jobs were created and configured with with different dependencies, In/Out conditions. etc (~300 is just the last step, as there are many more steps ahead it).

Thank you again,


RickM
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

As to your suggestion of have one job taking params from a text file, it wont fit the bill for my situation, because our jobs were created and configured with with different dependencies, In/Out conditions. etc (~300 is just the last step, as there are many more steps ahead it).



So create the job specification with a more flexible format - XML comes to mind. A Java object using the Command Pattern approach would also work.

Thats what I would do if this was my problem, and I would then experiment with the number of Threads in a Thread pool to which these jobs could be assigned, monitoring memory and CPU use.

Of course, if multiple computers were available then you could farm out the jobs with something like JavaSpaces, but for a single CPU there is no reason to have more than one JVM with multiple worker threads.

Bill
 
Ricky Murphy
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks guys. It makes a lot of sense. I will take it forward and hope that I can break the mold...
 
Something about .... going for a swim. With this tiny ad ...
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic