We have a couple of Tomcat servers. These use a JCIFS filter to authenticate to our domain controllers. We have a couple of Apache HTTPD servers. Each uses mod_jk to load balance between the Tomcat servers. We communicate through the Apache HTTPD servers. All of this is running on Windows.
This has all worked for quite a while now. However, the Apache HTTPD logs were getting quite big (access.log was 3GB), so we modified the logging to use Cronolog. We then restarted Apache HTTPD for the new logging settings to become effective, and all of a sudden we are getting 401 errors. The odd thing is, it only happens with Java clients (using HttpURLConnection). VB clients (don't ask...) or browsers have no problems whatsoever. There also is no problem if we connect directly to the Tomcat servers.
We have restarted Tomcat, no success. We have restarted Apache HTTPD, no success. We changed back the logging to default, no success. The only things we can see are the 401 errors in the Apache HTTPD access logs, and failures because of bad username / password combinations in our domain controllers' event viewer. The usernames are correct though, so it must be the passwords. We can't see what's being used though.
The JCIFS filter in the application's web.xml file; the JCIFS properties are set as Java properties to Tomcat.
The connectors in Tomcat's server.xml file (the rest is default); removing the tomcatAuthentication="false" parts also doesn't change a thing.
Part of Apache HTTPD's httpd.conf file:
We've already tried changing the JkOptions, especially the second one, but none of them works.
The workers.properties file:
The XXX in the server.xml and workers.properties files match.
Also, this entire problem disappears when we have only one server in our load balancer. However, doing that isn't really an option on the long term because we need to load balance for both performance and having a failover.
I've already looked at this problem all day (over 6 hours already), and I am all out of ideas. I have even tried to replace JCIFS with WAFFLE, but then Chrome and IE9 are no longer able to login at all (Firefox is though).
So, which genius can shed some light on this problem?
OK, so we decided to try and replace JCIFS. We tried WAFFLE and Tomcatspnego but neither allowed us to login at all, not even through browsers. So we ended up with Jespa, even though it's commercial.
Installation of Jespa went well. We got it running on a test environment, and it works from both browsers and Java code - as long as there's no load balancer in between. This time authentication fails even with one single server in the load balancer; browsers use Jespa's failback and Java code gives us the following exception:
The line the exception comes from indicates that some raw value is null.
Just as a notification, the issue has been found and resolved.
The NullPointerException with Jespa was caused by version 1.3.31 of mod_jk. This does not send the remote port, which was shown in the Jespa logs: "Insufficient getRemotePort value: -1". Upgrading mod_jk to 1.3.32 solved that issue.