If everything was public, everybody would have access to the internals of your class. This can be bad, preventing you from changing the implementation without breaking other code. If you keep everything as private as possible and expose only the publicly available 'interface' (mind, I don't mean
Java interfaces here, yet), you make sure all your users can only see these member, which gives you the liberty of reimplementing some stuff at some point (to make it better, faster, more error proof or simply different) and still be sure that clients of your class will continue to work as before (I am assuming that when you reimplement the internals, you remain compatible when it comes to the public behavior / contract). In Java you can go to the extreme and just define interfaces for clients of your code, and their implementations can be completely hidden and actually irrelevant to the clients. The general rule is to code against behavior, not implementation.
An example: ArrayList<
String> list = new ArrayList<String>(); // *
here you use list by its public methods. If you had access to the internal array and your code made us of it, it would be impossible (**) for the authors of this class to reimplement it, for example, using direct byte buffers or whatever (just an example). With protecting the internals, they are leaving themselves a door open for future reimplementations.
Having said that, there are languages that have public as the default, for various reasons. For example, Python doesn't even have anything less the public, you can only imitate private with weird naming conventions / trick; Scala, a very powerful and popular language for the JVM has public by default.
* This example can be written better: List<String> list = new ArrayList<String>() as in the 'always code against interfaces' rule, or even the list could be created by a factory or injected.
** This is not actually true - there exists a very powerful set of classes for Java - Reflection. With these classes you can access whatever you want, whenever you want, even private methods and fields, relatively easy. But that's outside of scope.
Hope this can be helpful.
