• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Tomcat + JCIFS + Apache + mod_jk + Java client == error 401

 
Sheriff
Posts: 22783
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We have a couple of Tomcat servers. These use a JCIFS filter to authenticate to our domain controllers. We have a couple of Apache HTTPD servers. Each uses mod_jk to load balance between the Tomcat servers. We communicate through the Apache HTTPD servers. All of this is running on Windows.

This has all worked for quite a while now. However, the Apache HTTPD logs were getting quite big (access.log was 3GB), so we modified the logging to use Cronolog. We then restarted Apache HTTPD for the new logging settings to become effective, and all of a sudden we are getting 401 errors. The odd thing is, it only happens with Java clients (using HttpURLConnection). VB clients (don't ask...) or browsers have no problems whatsoever. There also is no problem if we connect directly to the Tomcat servers.

We have restarted Tomcat, no success. We have restarted Apache HTTPD, no success. We changed back the logging to default, no success. The only things we can see are the 401 errors in the Apache HTTPD access logs, and failures because of bad username / password combinations in our domain controllers' event viewer. The usernames are correct though, so it must be the passwords. We can't see what's being used though.

Some settings:
The JCIFS filter in the application's web.xml file; the JCIFS properties are set as Java properties to Tomcat.

The connectors in Tomcat's server.xml file (the rest is default); removing the tomcatAuthentication="false" parts also doesn't change a thing.

Part of Apache HTTPD's httpd.conf file:
We've already tried changing the JkOptions, especially the second one, but none of them works.

The workers.properties file:
The XXX in the server.xml and workers.properties files match.

Also, this entire problem disappears when we have only one server in our load balancer. However, doing that isn't really an option on the long term because we need to load balance for both performance and having a failover.

I've already looked at this problem all day (over 6 hours already), and I am all out of ideas. I have even tried to replace JCIFS with WAFFLE, but then Chrome and IE9 are no longer able to login at all (Firefox is though).

So, which genius can shed some light on this problem?
 
Rob Spoor
Sheriff
Posts: 22783
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
OK, so we decided to try and replace JCIFS. We tried WAFFLE and Tomcatspnego but neither allowed us to login at all, not even through browsers. So we ended up with Jespa, even though it's commercial.

Installation of Jespa went well. We got it running on a test environment, and it works from both browsers and Java code - as long as there's no load balancer in between. This time authentication fails even with one single server in the load balancer; browsers use Jespa's failback and Java code gives us the following exception:
The line the exception comes from indicates that some raw value is null.

So we still have no idea where to go next...
 
Rob Spoor
Sheriff
Posts: 22783
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Just as a notification, the issue has been found and resolved.

The NullPointerException with Jespa was caused by version 1.3.31 of mod_jk. This does not send the remote port, which was shown in the Jespa logs: "Insufficient getRemotePort value: -1". Upgrading mod_jk to 1.3.32 solved that issue.

The error 401 was because of the notorious "Not a Type 1 message" error message (as found in the logs). With the help of Jespa's operator manual we were able to track down this problem to cookies. We did not use cookies at all, so the problem should actually have occurred from the start. The solution is surprisingly simple - we just added the following lines to our code:
This uses an in-memory cookie store and accepts all cookies. The cookie store can be replaced as desired (like use persistent storage), but the default cookie policy will not work. The load balancer's domain and the cookie domain from Tomcat will not match, and therefore the default (accept from original server) will reject the cookies. A different cookie policy can be used, as long as it allows cookies from the Tomcat servers. Since our application only connects to the load balancers the accept all policy is good enough.
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic