When they first come to interview, don discuss about the job scope first, give a test paper for them to work on it
1) if your company software is already up there and just need a programmer to write Java code to change some Change Request
and your company just need Java programmer who don't know Object-Oriented and code like a C programmer
a) a question on simple logic ( for...loop, if...else, switch...case, etc.)
b) give a Class with static and non-static method, ask them to write their own class to call the method
2) if your company software development follows good practice, design pattern, architecture, layering
a) prepare MCQ which similar to those SCJP , and only test those question related to what your company need
b) get a technical strong staff to interview that guy on all those concept (and HOW did they use it practically in past experience)
... what I am trying to say is ask and test them on what your company WANT and NEED
I have go through companies which give those MCQ test up to SCJP standard, test OO concept, and talk about MVC, multi-tier, layering ... but end up their code using Map/Vector as a Class to store attribute and value (argue that it is for flexible), getting to see presentation layer code call stored procedure directly, business logic in controller .... So, if your company doing all those things (which I don't want to argue whether it is right or wrong), you need to discuss all this with the candidate, and find out whether they feel that they are suitable for this ... but not trying to hide it.