aspose file tools*
The moose likes Other Open Source Projects and the fly likes Apache Seems to Have Gone Deaf Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Products » Other Open Source Projects
Bookmark "Apache Seems to Have Gone Deaf" Watch "Apache Seems to Have Gone Deaf" New topic
Author

Apache Seems to Have Gone Deaf

Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

So our test server was running Apache with Shibboleth and Tomcat. (For running Liferay). I had Web SSO set up to log me into the test Liferay and all was peachy...

Until the server crashed and got restarted.

I don't know what caused the crash but now Apache seems to be having issues. I've shut down Tomcat and Shibboleth to test just the Apache by itself and nothing. Unable to connect. Tried getting a response from Apache using the command line:

telnet localhost 80

and connected, so I sent the command

HEAD / HTML/1.0

and got this back:

HTTP/1.1 500 Internal Server Error
Date: Tue, 01 Nov 2011 18:11:49 GMT
Expires: 01-Jan-1997 12:00:00 GMT
Cache-Control: private,no-store,no-cache
Content-Length: 960
Connection: close
Content-Type: text/html; charset=UTF-8

Now, I know the machine has the appropriate ports open because I was able to access Liferay through Tomcat when Tomcat was running... So the machine is live, it's listening on port 80, but it seems Apache has decided to sit out. I've restarted httpd with no effect.

Looking at the error logs, I'm not finding any errors of any kind. Am I missing something stupidly obvious?

I've made NO configuration changes since the crash, and I've checked all my config files anyway just to be sure they weren't corrupted.
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
Should it not be
HEAD / HTTP/1.0
?
Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

Karthik Shiraly wrote:Should it not be
HEAD / HTTP/1.0
?


Yes, thanks. That was a typo on my part here. On the server I did enter "HTTP."
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
If httpd logs are not giving any clue, increase the LogLevel to debug (in httpd.conf) and see if there's anything unusual.
What OS is this? Assuming OS is *nix, check if syslogs hold any clues at the time of crash and later.

I don't think httpd is hanging - it's just not giving any HTTP response.
What file should be served at that path "/"?
If it's a PHP file, I've seen this happen if there's some syntax error or runtime error in PHP files. That error is captured in the PHP log, but if PHP logging is turned off, then it's difficult to find out. If using PHP, check that php.conf has log_errors set to on and error_log pointing to a writeable path.

Other things to check:
- file permissions - the apache process should be able to read the file and its parent directories
- not enough disk space for logs, caches, etc
- if using mod_jk, check the log file specified by JkLogFile in mod_jk configuration
Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

Hey Karthik, thanks for the reply.

Karthik Shiraly wrote:If httpd logs are not giving any clue, increase the LogLevel to debug (in httpd.conf) and see if there's anything unusual.
What OS is this? Assuming OS is *nix, check if syslogs hold any clues at the time of crash and later.


This is CentOS. Looking at the syslog in this case didn't help because it looks like there are no log files at all that remain from before the crash. I don't know why this is. I'll be meeting with the sysadmin later. Maybe he knows.

Karthik Shiraly wrote:
I don't think httpd is hanging - it's just not giving any HTTP response.
What file should be served at that path "/"?
If it's a PHP file, I've seen this happen if there's some syntax error or runtime error in PHP files. That error is captured in the PHP log, but if PHP logging is turned off, then it's difficult to find out. If using PHP, check that php.conf has log_errors set to on and error_log pointing to a writeable path.


There's a generic index.html file there. (In the DocumentRoot directory) for testing purposes. Prior to the crash, I used mod_ajp to connect to a Tomcat servlet container.

Karthik Shiraly wrote:
Other things to check:
- file permissions - the apache process should be able to read the file and its parent directories
- not enough disk space for logs, caches, etc
- if using mod_jk, check the log file specified by JkLogFile in mod_jk configuration


File permissions and the apache user are unchanged from before, and the disk has plenty of space. We're not using mod_jk.

At this point, all I really want to do is get the basic index.html page to display in the browser to use as a starting point and I can't even get that.

Interestingly, after a server reboot I ran the HEAD / HTTP/1.0 command again and got this:


HTTP/1.1 200 OK
Date: Wed, 02 Nov 2011 15:54:19 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Thu, 11 Aug 2011 18:55:45 GMT
ETag: "24f2c4-6f-4aa3f58200e40"
Accept-Ranges: bytes
Content-Length: 111
Connection: close
Content-Type: text/html; charset=UTF-8


But still no response when using a web browser.
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
That's very odd.
What HTTP code does the browser receive, if any?
I notice the telnet was done from localhost. Is the browser on another machine, and if so, is httpd listening on that LAN IP address?


Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

The browser receives no code at all.

I also tested a GET request over telnet to make sure it was finding the correct DocumentRoot folder ans I got the HTML output to the terminal, so it's definitely running and finding the file.

The browser is indeed on another machine, and httpd is configured to Allow from all on the DocumentRoot folder.
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
So, browsing from the client machine gets a "could not connect" kind of browser error screen, but telnet from the same client machine gets a response?! It gets odder .
Is it a browser proxy issue or something like that, independent of the actual apache crash problem? Does clearing the browser cache help? Sorry, at this point, I've run out of ideas.
Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

Well no, when I use telnet it's when I'm connected to the server through ssh. That's why I can use localhost.

I don't think Apache is crashing... I think it's just not receiving the http request. No configuration has changed from before so I think it may be a corrupted file somewhere...
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
Ah ok, so port 80 is not reachable from remote machines. I'd look at "netstat -anp" output to ensure it's listening on that LAN IP too or on all IPs (0.0.0.0:80).
If that's also ok, then check if some firewall configuration has changed (/sbin/iptables).
If it was a corrupt file, then I think it would have manifested in the telnet output too, but that's not happening.
So perhaps it's an inadvertent configuration change / update done by sysadmins.
Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

If any of those weren't open, wouldn't it also prevent a request from reaching Tomcat? Or am I misunderstanding something and Tomcat circumvents this?
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
Yes you're right - if apache is fronting tomcat on port 80 and there's a firewall problem, the liferay path should not have worked either. Guess that's not the problem then - I missed your statement about having tried tomcat after the crash.
Guy deLyonesse
Ranch Hand

Joined: Apr 12, 2011
Posts: 200

SUCCESS!

It looks like what changed was the IP tables. The sysadmin had suggested I look at it before but at the time I had another idea in mind of what I expected to find, but on your advice I looked again and this time I saw it...

Port 80 and 443 are being redirected to 8080 and 8443, respectively... and Apache was only listening on 80 and 443. Tomcat responded because it IS listening on 8080 (and then doing a redirect... The way I had my configuration with Shibboleth, Tomcat would only be connected to through 8009 per the ajp config.)

This is not how it was prior to the crash, as far as I know.

To test this, I set Apache to listen on 8080 and 8443 and sure enough, it worked! So I re-enabled Shibboleth and that worked too. So now I need to get the iptables set back to the way they were so I can put my configuration back the way it was and all will be well.

Thanks very much for your help!
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 513
    
    6
You're welcome! These environmental issues can be major time wasters - good that this one got solved relatively quickly.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Apache Seems to Have Gone Deaf