In both JAX-RPC and JAX-WS message header and payload validation would typically happen in a MessageHandler. If validation fails the handler would initiate the SOAPFault and the service application will never see the request.
The SOAPHandler receives a SOAPMessageContext. With SOAPMessageContext.getMessage().getSOAPBody() you can get the javax.xml.soap.SOAPBody which implements org.w3c.dom.Element which ultimately gives you access to the SOAP payload (XML document) which you should be able to validate. Raise a javax.xml.ws.ProtocolException if the validation fails. At this point fault processing begins and the handleFault method is called on the handlers that have already processed the in-bound request in reverse order. That means that your validation handler is the first handler called - so you assemble the fault response in its handleFault method (if it was the one that threw the ProtocolException in the first place). Alternately you could just simply build the fault in handleMessage and give it to the SOAPFaultException that you throw (example).