I have an IIS 7.5 talking to Tomcat 7 via Jakarta Isapi Redirector.
I have set up IIS to only allow Windows Authentication. I am only using the Default Website in IIS. So (I am assuming) that all request will get redirected to Tomcat. And when the IIS gets the request, my application is displayed without the user having to enter ID and password. SSO is working. Let's call this webapps smsso.
I have another Tomcat webapps where it is created for users that does not have any domain/windows account (/smxsso). That means, IIS should not ask for any windows credentials.
So, I allowed Anonynous Access in IIS. But then it breaks my webapps that is meant for SSO. The application now display it's login page (where before it does not).
My question now is: If I enable Windows and Anonymous authentication in IIS, how can I tell Tomcat that /smsso/ will use windows authentication and /smxsso/ will use anonymous authentication?
This depends somewhat on how your Tomcat app gets secured. If the webapp has its own user-designed login code, you have a challenge on your hands. Whatever solution your come up with will be yours and yours alone, just like the app's login code.
On the other hand, if the webapp is delegating login to the container using the J2EE container-managed security system, then the login (or lack of it) becomes the responsibility of whatever Realm implementation you use.
There is at least one Realm module that will work with Windows User security. There is also at least one Realm module that allows you to combine Realms so that for example, Windows (LAN) security may be combined with a more general solution such as a database or LDAP service.
Although before getting too creative in that regard, I should observe that you're probably better off letting IIS proxy ALL user requests targeting Tomcat, and not just some of them. Or if you have reasons for not using IIS for the non-LAN users, use something like Apache, which can present its own security interface while simultaneously eliminating some of the problems with Tomcat connecting to the open Internet directly. If you use IIS, I would hope that there's a way for IIS itself to manage the login process for the non-LAN users. Since I haven't worked with IIS in many, many years, I can't say for sure about that, however.
An IDE is no substitute for an Intelligent Developer.