File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Distributed Java and the fly likes Complex high performance message queues Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login

Win a copy of Java Interview Guide this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Java » Distributed Java
Bookmark "Complex high performance message queues" Watch "Complex high performance message queues" New topic

Complex high performance message queues

Dave Peterson

Joined: Jan 24, 2009
Posts: 4
I have an extremely high performance high scale high available problem. It's sort of combining producer/consumer, broadcast/subscribe in a very high volume basis. Think 10k transactions per second with responses needed in 50msecs in a 24x7x365 up time scenario.

I have a pool of producers spread across multiple servers that get work. Ideally each producer would get a unit of work and broadcast the job out to all consumers of different types.
1. A Producer instance broadcasts job to consumer type A, consumer type B, consumer type C spread across multiple servers.
2. Consumer type A, B, and C work on the problem and respond with their individual results.
3. There will be many instances of consumer type A (or B or C), but only 1 instance should work on any one job.
4. Producer instance waits for results from A, B, and C and then picks the most optimal result from the work done by A, B, and C. If A, B, or C are slow to respond, they lose (expiry maybe?).
5. Producers and consumers will all be in multiple servers (VIPs).

What are the best technologies to consider for this?
Jimmy Clark
Ranch Hand

Joined: Apr 16, 2008
Posts: 2187
MQ Series is the right tool for this. IBM has a talented group of software engineers. They would certainly be able to help you determine which technologies are best suited for the requirements and also how to design a solution that works.

Good luck!
William Brogden
Author and all-around good cowpoke

Joined: Mar 22, 2000
Posts: 13037
That really sounds like a JavaSpaces problem to me, given the transaction requirement.

GigaSpaces has both developer and commercial versions.

With that volume, will network speed be the limiting factor? Let us know what you find out!

I agree. Here's the link:
subject: Complex high performance message queues
It's not a secret anymore!