Simulate HTTPS in Tomcat connector or Apache config?

[Richard Levy]

Hi all,

First of all, sorry for the long post, but the reasoning and architecture requires explanation to get the full understanding of the situation.

We are currently migrating our website from Oracle 10g to Apache 2/Tomcat 6.0.29. The site itself is a set of pages where customers select what they want to purchase, then once they're happy, it switches to SSL to gather their customer details and take payments.

The current setup involves two intelligent load-balancers that terminates SSL then hands over to Oracle 10g. The 10g stack has custom Oracle versions of Apache which have configuration options not found in standard Apache and appear to have made configuration simpler. For instance, there is a "simulateHttps" directive that you can give to a virtual host in Oracle Apache which does not exist in the standard version. Using that option, we simply have 2 virtual hosts defined in Apache, one for http traffic and one for https traffic (with simulateHttps set to on). They both forward to OC4J and it works fine.

With the new architecture we have two load-balancer which round-robin to two Apache servers. These servers connect to two Tomcat servers which are configured with a mod_jk loadbalancer using session affinity. Through config we have separated HTTP & HTTPS traffic. We have done this because the application needs to identify when it is secure, which we do by simulating HTTP using settings in the connector in the Tomcat server.xml. This is detailed further down.

Obviously we want to keep the same functionality on the Apache/Tomcat stack, and also retain the termination of SSL at the load-balancer, but are having trouble when the application switches to HTTPS. It works fine if we have a single Tomcat instance running, but once the 2nd Tomcat instance is enabled, moving to HTTPS fails because a new session is created.

Our configuration is as follows:

Apache

We have 2 Apache servers running - they are served in a round-robin style by the load-balancer. They recieve 2 types of traffic, non-secure and secure, but the secure traffic is NOT encrypted (SSL terminated at the load-balancer).

httpd.conf:
JkWorkersFile conf/workers.properties Listen 5555 Listen 6666 NameVirtualHost *:5555 NameVirtualHost *:6666 <VirtualHost *:5555> ServerName http://www.mysite.com:80 JkMount /* loadbalancer </VirtualHost> <VirtualHost *:6666> ServerName https://www.mysite.com:443 JkMount /* loadbalancerSSL </VirtualHost>
We are using mod_jk to connect to Tomcat

workers.properties

		worker.list = loadbalancer, loadbalancerSSL, status
		worker.tomcat1.type=ajp13
		worker.tomcat1.host=192.168.1.36
		worker.tomcat1.port=8009
		worker.tomcat2.type=ajp13
		worker.tomcat2.host=192.168.1.85
		worker.tomcat2.port=8009
		
		worker.tomcatSSL1.type=ajp13
		worker.tomcatSSL1.host=192.168.1.36
		worker.tomcatSSL1.port=8010
		worker.tomcatSSL2.type=ajp13
		worker.tomcatSSL2.host=192.168.1.85
		worker.tomcatSSL2.port=8010
		
		worker.loadbalancer.type = lb
		worker.loadbalancer.sticky_session = 1
		worker.loadbalancer.balance_workers = tomcat1,tomcat2
		worker.loadbalancer.session_cookie = JSESSIONID
		
		worker.loadbalancerSSL.type = lb
		worker.loadbalancerSSL.sticky_session = 1
		worker.loadbalancerSSL.balance_workers = tomcatSSL1,tomcatSSL2
		worker.loadbalancerSSL.session_cookie = JSESSIONID
		
		worker.status.type = status

As you can see, Apache can receive on either 5555 or 6666 dependant on the source of the traffic. If it's non-secure, it uses the loadbalancer called "loadbalancer" which uses two Tomcats listening on port 8009 - this is a connector in Tomcat that is not secure. For traffic on 6666, "loadbalancerSSL" is used. This is the same two Tomcats but on port 8010 - this is a connector that believes it is secure. Tomcat config is detailed below.

Tomcat

We have 2 Tomcat servers. They too recieve two types of traffic, non-secure and secure, as passed on from Apache.

server.xml Tomcat 1:
<!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port="8009" protocol="AJP/1.3" redirectPort="8010" maxThreads="200"/> <!-- This is the secure connetor, although its not using SSL, it's just secure traffic --> <Connector port="8010" protocol="AJP/1.3" connectionTimeout="20000" scheme="https" SSLEnabled="true" secure="true" maxThreads="200" proxyPort="443"/> .... <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
server.xml Tomcat 2:
<!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port="8009" protocol="AJP/1.3" redirectPort="8010" maxThreads="200"/> <!-- This is the secure connetor, although its not using SSL, it's just secure traffic --> <Connector port="8010" protocol="AJP/1.3" connectionTimeout="20000" scheme="https" SSLEnabled="true" secure="true" maxThreads="200" proxyPort="443"/> .... <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">
As can be seen, the only difference between the two Tomcat configs is the jvmRoute parameter on the Engine config.
The connector on 8009 is a vanilla connector configuration. On 8010, SSL related parameters are set so that it believes it is secure.

What's happening

If I have a single Tomcat running, I can progress all the way through the site, go secure, retain the session and get to the end of the process. As soon as the 2nd Tomcat in introduced, when it goes secure, the site returns to the first page. I believe this is because sticky sessions fails and the requests ends up on a separate server, hence creating a new session. The application detects this and sends the user back to the first page in the process (it needs the data in the session at this point and as there is none, redirects to the start).

I think that sticky sessions fails because the process behind it uses the jvmRoute value from the Tomcat server.xml to append to the end of the JSESSIONID cookie. When the app goes secure and Apache receives traffic on 6666, it starts to look for tomcatSSL[1/2] but none of the tomcats use that jvmRoute and hence it will not be part of the JSESSIONID, so it will not stick to the same server. The reason that it works with one server is because it only has a single server to send it to, therefore will always find the session.

If this is the case, we may have approached this incorrectly. Separating non-SSL and SSL traffic all the way down to Tomcat may not be the answer and we may need to use something in Apache to modify request and response, in the same way that I assume "simulateHttps" worked.

Has anyone else setup a site using this sort of architecture? I can't imagine that this is uncommon in the enterprise?

Any help is greatly appreciated!

Many thanks
Rich

[Tim Holloway]

Mixed secure/insecure webapps are common as dirt. However, complicated solutions like this, no, load-balancing or not.

I'm not sure what these Oracle (alleged) enhancements to Apache are good for, but it's for certain that the bog-standard Apache/tomcat configuration doesn't need them, as I am getting along just fine without them, and I'm using the stock Red Hat architecture that Oracle pilfered to create their "Unbreakable Linux" (probably TM).

This is one of the situations where I MIGHT be able to tell, if it was worth it to me to kill a tree and draw diagrams with circles and arrows on it, where you're going overboard. However, frankly, I charge money for that much work.

But I have a suspicion, and my suspicion is that you're not understanding the mechanism that Tomcat uses to switch into SSL mode - which is the SSL "redirect port" specifications in server.xml.

Actually connecting Apache and Tomcat can be done in one of 2 primary ways. Traditionally, it was done via mod_jk, which became mod_jk2, then mod_jk3, which was then folded back into mod_jk2 and abandoned. The newly-recommended way is to use Apache proxy capabilities instead. Either way works, although I find the log files get a little noisy when using the proxy method, because spammers attempt to use the reverse proxy as a spam relay (and will succeed if you don't set things up correctly).

The key factor here is that both SSL and non-SSL channels must be channeled - typically port 80 to Tomcat's 8080 and 443 to Tomcat's 8443 for proxying, although mod_jk will use its own private channel (typically 8009).

As I mentioned before, just because the ID of the session changes, doesn't mean that a whole new session is created. However, to ensure that sessions stick, you do need to make sure that you have the backend Tomcat clustering options configured properly. Aside from purely practical concerns, load-balancing is something that normally also expects server failover capabilities as well.

[Richard Levy]

Hi,

We solved the problem in the end - well, with help from the tomcat-users mailing list.

It came down to my lack of understanding about the AJP connector. I'd assumed (incorrectly) that it was pretty dumb and we needed to separate out the SSL traffic all the way to Tomcat (hence the 2 connectors). In reality, the AJP connector is pretty smart and retains information about the request. We simplified our configuration and now it works.

Full details can be found here : http://www.mail-archive.com/[e-mail]/msg88208.html.

Thanks for your help.

Rich