This week's book giveaway is in the OCMJEA forum. We're giving away four copies of OCM Java EE 6 Enterprise Architect Exam Guide and have Paul Allen & Joseph Bambara on-line! See this thread for details.
So my suggestion is to use a Flash based file uploader. Swfupload support this kind of validations in client side itself. Don't know if it is possible with JavaFX. If possible you can give it a try.