I have understood it as follows (please correct me if I am wrong!):
When a MDB is involved in a transaction which includes receiving a message, rolling back the transaction will result in the message being re-inserted into (or not removed from, in the first place) the queue. Inserting the message into the queue is done in another, earlier, transaction.
Remember that messages to MDBs do not go directly from the sender to the MDB, but always via a queue.
EJB 3.1 Specification, chapter 13 (with sample scenarios in section 13.2).