And now it works! Somehow the .JSP which processes error 401 was changing the response in some way such that the authentication wouldn't work. It was obvious what the problem was when
it was forcing the system to use Basic authentication , but I don't understand why it would continue to fail when I had modified the 401.jsp to use digest authentication.
subject: Tomcat 6.0.26: Manager with DIGEST authentication