It depends on the webapp server you are using and on how it is configured. Given the "wrong" settings, tweaking a webapp on the fly won't work at all.
Changing web.xml is potentially changing basic characteristics of the webapp. Among other critical architectural specifications, web.xml determines how URLs are routed to servlets, what the security rules are, and whether or not selected URLs are process via secured (SSL/TLS) network channels.
Because of this, the only really safe way to update web.xml is to halt the webapp, rebuild all digested information in web.xml and restart it - effectively re-deploying it. Otherwise, at best what you'd be doing is the equivalent of changing the tires on a car while it's racing down the Autobahn.
Customer surveys are for companies who didn't pay proper attention to begin with.