The container will not rollback the transaction if an application exception is thrown as it is assumed that the client expects that exception (can recover from it). As an application exception is a checked exception, check what you are throwing. Could it be SQLException? If so, you probably won't want to throw this, in which case
you should wrap it in EJBException and throw EJBException.
If you do throw an application exception but decide that it makes no sense to continue with the transaction, then you must first invoke setRollbackOnly() followed by the throwing of that application exception. setRollbackOnly() will force the container to rollback the transaction.