This week's book giveaway is in the General Computing forum. We're giving away four copies of Arduino in Action and have Martin Evans, Joshua Noble, and Jordan Hochenbaum on-line! See this thread for details.
If you have in your deployment descriptor, this will override the corresponding annotation for the enterprise bean. So in your case, the EJB container is handling the transaction, even if your MDB is annotated with A transaction is opened by the container immediately before any call to onMessage, and commited/closed immediately after, unless a system exception is thrown, or the setRollbackOnly method is called on the EJBContext.
Using try/catch/finally does not imply the transaction is being handled by the developer, only that you're dealing with exception coming from your business layer (or anything invoked by your MDB).
To handle yourself the transaction, you have to remove the ejb-jar.xml fragment for this MDB (or set the transaction type to BEAN), and use the TransactionManager interface to begin(), commit() or rollback() the transaction in your code.