The worst performance comes when you have LAZY on data that you will be using. So let's say your database has a USERS table and a related ADDRESS table. You retrieve a list of users, and you want to print out their addresses. Hibernate will do one query for the list of users and then one more query for the addresses for each of the users. That's a total of N+1 queries, where N is the number of users. Also know as the N+1 problem, this represents an epic failure in your mappings.
On the other hand, you could put EAGER on the addresses, but what if you rarely actually care about that data. The Hibernate query for users will include a join with the address table, which of course will be slower than a simple query of the users table. If you don't access the addresses, then you've wasted the effort to retrieve them.
Now, it is possible to write Hibernate (and JPA) queries to explicitly join in the addresses where you want them. That will override the LAZY load in the mappings. However, if you forget to do it, then you get the N+1 problem again.