Well objects are like persons in your class and access modifiers are like some specialties to the properties of objects. Say, take a name having # of vowels etc but person being an object itself, it may not possess these.
Of course, how do you get access to a person? Through his roll number in a class --- nothing but a reference (more like a pointer) that knows how to reach an object!
Now when it comes to 'access modifiers' -- a means to control the access, references are also members of a class in which it is declared. so they are very well eligible to hold those.
The example what Campbell gave explains you very clearly.
Bert Bates and Kathy Sierra have come up with what I think is a pretty good analogy. Think of the object as a TV, and a reference as a remote control. you never touch the TV itself, only the remote. you use that to access the functions of the TV.
There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors