This week's book giveaway is in the Mac OS forum. We're giving away four copies of a choice of "Take Control of Upgrading to Yosemite" or "Take Control of Automating Your Mac" and have Joe Kissell on-line! See this thread for details.
So my suggestion is to use a Flash based file uploader. Swfupload support this kind of validations in client side itself. Don't know if it is possible with JavaFX. If possible you can give it a try.