I presume by Driver you mean something like com.databasecompany.drivers.Driver. If you look up the static Class#forName(String) method, you find it returns the Class<?> object of that name. It is a little like a static counterpart to the instance method Object#getClass(). Now, to get a Class<?> object, you have to have the Class in JVM memory, so if the class of that name has not yet been loaded, it loads it into JVM memory. And whenever you load a class into memory, its static members are loaded too, and any static initialisers are run (you can tell that from the list of Exceptions shown in the link.
Most JDBC driver classes have a static initialiser which sets up the connection to the database management program. Running Class.forName(...) at the start of the program will call that static initialiser and connect you to the database.