• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

EJB Overhead and Performance

 
Author
Posts: 6055
8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We just finished the first phase of our project. We built a basic financial trading system. It uses JMS to communicate between rich Swing clients and an EJB backend. The software looks good, but I'm a little concerned aout its performance
We don't implement encryption (it's fake money in a classroom, so we're not worried about hackers), so that saves us some expensive processing. We also aren't totally concerned with crashes. We'll have the holdings in the database, and if the last transaction didn't quite work (e.g. maybe during a failure, someone bought stocks but never paid for them, eh, we don't care). the bulk of the processing is simply handling and crossing orders. Everytime we get an order, here's what we do:
1) Get all holdings of the user, and mupltiple the quantity x current value per share x margin ratio (it's clightly more complicated then this, but this is a reasonable approximation).
2) Do the same as in step 1, but for outstanding limit orders.
3) If the numbers pass some check, we let the order go through (otherwise we jump to step 7, with a "not allowed" message)
4) We get the set of limit orders, cross the incoming order with them.
5) We update all holdings in the database.
6) Send an announcement about the cross to all users.
7) Send updates to each user involved in the transaction.
8) Recalc the margin for the order placer (steps 1 & 2), based on their new holdings and limit orders.
We take orders both from users and from computers traders who run in a background thread on the server and place orders based on a formula.
We're running BEA Weblogic 8 on Win2000 on a 2.4GHz, 1G RAM machine. The database is Oracle 9i (I forget the specs, but its a slightly better machine).
Unfortunately, this is running far slower then we need.
We used to keep all limit orders in the database, because that took care of the locking for us. It was too slow. We now copy the limit orders to memory, maniuplate them during the order crossing, and then write them back to the database if they are new or changed (as opposed to reading each limit order as needed).
I recognize that step 8 duplicates work, and can spend some time speeding it up, but after looking at the profiler, that's not a huge performance impact. According to OptimizeIt, about 60% of the time is spent in WebLogic classes. It doesn't give me any more detailed information.
I can't believe that WebLogic classes are really using that much time. Does this include complied bean classes that are my code, but automatically generated? Any way to get more detailed information?
Roughly speaking, how much overhead do EJBs produce? My gut feeling (yes, I know, dangerous when looking at performance) is that the calculations aren't killing us; that's not to say they aren't significant, just that something else is sucking up time, but I can't see it in detail.
Does anyone have experience with JMS-driven EJB systems? Anyone have experience with financial applications like this? Or any systems which seem similar? I'd like to hear some rough numbers in terms of application support and load factors and machines used, and such. Heck, most anything will be helpful, as I'm feeling kinda in the dark here.
Thanks for reading all this!
--Mark
 
Ranch Hand
Posts: 1561
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The first thing that pops up into my mind is.. did you have to use EJB's ??
How many *concurrent* users will you be expecting?

...and if the last transaction didn't quite work (e.g. maybe during a failure, someone bought stocks but never paid for them, eh, we don't care)...


so you're saying transactions is not that important in your system?
hmm...
 
Andres Gonzalez
Ranch Hand
Posts: 1561
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I just happened to find this URL
http://www.precisejava.com/javaperf/j2ee/EJB.htm#EJB142
at the end, in the Common tuning practices for EJBs:

-Choose normal java object over EJB if you don't want built-in services such as RMI/IIOP, transactions, security, persistence, resource pooling, thread safe, client state etc..

Here transaction is also mentioned... That's why I was wondering about...
 
Mark Herschberg
Author
Posts: 6055
8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We are expecting 100-200 concurrent users. We'll also be supporting 100 some different securities, of 7 different types. Lots a simultaneous calculations. And of course, it seemed like we'd keep all orders in a database, so we'd have lots of DB pooling issues. Would we prefer transactions? Sure. It's be nice that if something goes wrong, the server is in a clean state, so that was a motivating factor, too. However, given where we are, we would trade safty for speed.
Looking back, it seems that EJBs may not have been the best choice. Still, it seems like the added cost of building/buying tools to do the overhead provided by EJBs seemed relatively expensive, compared to the thought of just buying a faster machine.
I'll check out those links. Thanks.
--Mark
 
Sheriff
Posts: 5782
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Here's a great favourite of mine - EJB 101 Damnations. It examines the use of EJBs under a magnifying glass and elaborates on why EJBs are not made for every J2EE implementation.......
 
Mark Herschberg
Author
Posts: 6055
8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thinking about it more, one of the things that really killed us is not having the database access fast enough. The project sponsors want 100 students to all trade the same stock at the same time, i.e. 100 orders all arrive together, and must be processed within 1 second. (Personally, we think this is really peak and not average load, given that they only expect 100 students in a given class, but they feel pretty adamant about it.)
Now here's the tricky part. We had to move the limit orders from the database because it was too slow. However, we can't simply have multiple beans processing orders at the same time. Specifically, if we get two orders at the same time in the same security, we must process them sequentially. (If they are in different securities, they can be processed in parallel.) So, to do this, we effectively do the following:
1) Put all limit orders for the same security into a queue.
2) Every second assign an OrderProcessorBean instance to a queue, to handle the orders for that security. When the bean gets assigned to the queue, it grabs the limit order book, stored in memory, from the JNDI tree.
3) When its done processing orders, it updates the limit order book and rebinds it to the JNDI tree.
It feels like what's wrong is the fact that we need to serialize some of the processing (for orders in the same security), and needed to create our own locking mechanism because we couldn't rely on the database and meet our performance goals.
--Mark
 
Ranch Hand
Posts: 580
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Why do trades on the same security have to be processed in the order they are received ? Is this a hard requirement ?
D.
 
Mark Herschberg
Author
Posts: 6055
8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Absolutely!
Suppose you and I both wanted to buy 1,000 shares. if my order comes in first, it should be processed first. The reason for this is because orders have price impact. Imagine if instead of 1,000 shares, it's 10,000,000 shares. If someone buys 10,000,000 shares, the price of the stock will go up, since there is huge demand. So an order for 1,000 just before it will pay less per share then 1,000 just after it.
Actually, this isn't the key issue. We use MDB's, and if two messages arrive one just before another, but because of thread swapping, the get processed out of order, who can tell? (In other word's, where exactly is the line to determine who is first). The key issue is that orders are serialized. I should finish crossing all the shares of one order, before crossing the shares of the second order. Again, you can imagine two orders of 10,000,000 shares each, such that each has price impact. The traders will pay different amounts depending on whether the orders cross, (A then B), (B then A), or (A and B interleaved). However, the major exchanges don't allow the third option, so our simulation can't either.

--Mark
 
Don Kiddick
Ranch Hand
Posts: 580
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So you are saying if Trader A makes an order A, 1 milisecond before Trader B makes an order B then order A should be filled entirely before order B ?
Don't orders have price associated with them anyway ? E.g. I'll buy 50 LYHOO at $23. So if you processed the two orders simultaneosly then the effect would be that both would have less probability of being completely filled. So in the example, order A & B would be 90 filled. This seems fair enough to me.
Anyways it's your system, you probably know best !
D.
 
Mark Herschberg
Author
Posts: 6055
8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes, exactly.
"buy 50 LYHOO at $23" is what's known as a limit order. If the lowest sell price is $24, then your order won't go through. It'll just sit there until either someone does offer to sell at $23, or the market moves away from it, and you cancel the order. The other type of order is a market order, "buy 50 LYHOO," which buys it at the best current price. (BTW, some of these orders all "all-or-nothing" orders which must all be filled at once or skipped in the crossing process.)
You are correct that the ordering of limit orders miliseconds apart is unlikely to have significant consequences (unless a market order comes between them, which would hit them). But for market orders, there's a huge difference.
--Mark
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic