Hello Doug,
You are right - there is no tangible criteria for discerning between 'when to use an exception or when to use a return value'. The issue is a result of various literature which uses terms such as "exceptional condition", "unrecoverable error" or "unforeseen developer mistake" - all of which cannot be formally defined.
If you view exceptions merely as contractual signalling in that the method that was invoked is unable to return a type successfully, then it quickly becomes apparant that there are a few problems with
Java exceptions. One of which is the implicit unrolling of the call stack.
Imagine the following simple example:
One could read that contract loosely as "the method will return an int, unless it can't, in which case, it will signal to the caller accordingly". The issue now is that the call stack has unrolled and the caller is unable to act in a way to recover. It is also problematic that exceptions in Java are defined as classes, since quite clearly, they are merely metadata on a contract, and nothing more - as opposed to the definition of a class.
You will find much literature that attempts to explain the merits or otherwise of Java exceptions, and in particular the "checked exception versus unchecked exception" debate, which started a few years ago and is unresolved. As far as I am concerned, it will remain forever unresolved until someone points out that the debate rests on a flawed premise. I can only suggest that you do not buy into what this literature says (or what anyone says including myself) without thinking for yourself.
I hope this helps.