We are talking about Object fields here, not primitives. When you have a field declared as final, the reference will not change. It will always point at the same Object. However, if the Object is not immutable, the methods on it (or the Object's visible fields) can be used to change the Object itself - it is the same Object, but its properties have been changed. With an immutable Object, no method call would change the Object's properties, and no non-final fields would be visible.
Does that take away any shred of clarity you still had on the subject?