Actually, neither.
You are correct in saying that opening a new connection for each operation is inefficient, but if there was only one connection per servlet it would tend to choke the performance of busy servlets (that get hit more often).
Likewise, creating one for each session ends up assigning resources which largely remain idle (and is therefore a waste) and limits the scalability of the app. (if you want to design for 100 concurrent users, how many databases can be expected to handle this?)
The solution is that many (?)
J2EE complient Application servers provide a connection pool. When the server starts up, it creates a set of connections that it maintains separate from the servlets. Each servlet can then ask for a connection from a common pool. If an Application server doesn't provide Connection Pooling it isn't too difficult to implement a simple version yourself.
Dave.