ClassLoaders provide part of Java's security model. A class is identified not only by its name, but by the ClassLoader that loaded it, and this ClassLoader can reliably be used to grant privileges to the class based on where it came from. java.* classes loaded from rt.jar are allowed to do things that, let's say, a downloaded applet class would never be allowed to do.
Many JVMs contain far more than three classloaders -- a web container includes at least one for each webapp, for example.