wood burning stoves 2.0*
The moose likes BEA/Weblogic and the fly likes Issue with WeblogicServer Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Products » BEA/Weblogic
Bookmark "Issue with WeblogicServer" Watch "Issue with WeblogicServer" New topic
Author

Issue with WeblogicServer

Piyush Kumar Sharma
Greenhorn

Joined: Jan 20, 2009
Posts: 1
Hi All

I am getting below mantioned error while stoping the webserver

Failed to connect to t3://MYSERVER:8098: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination

Please help me out.....
Amit Kumar.
Ranch Hand

Joined: Feb 27, 2009
Posts: 32
We get this when there are any Connection Reset Errors. Check your connection pool and data source.
Amit Ghorpade
Bartender

Joined: Jun 06, 2007
Posts: 2716
    
    6

"Amit Kumar Hcl " please check your private messages for an important administrative matter. You can check them by clicking the My Private Messages link above.


SCJP, SCWCD.
|Asking Good Questions|
Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977
Piyush Kumar Sharma wrote:Hi All

I am getting below mantioned error while stoping the webserver

Failed to connect to t3://MYSERVER:8098: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination

Please help me out.....


Usually that's exactly what means it can't resolve MYSERVER:8098 as a weblogic server instance(i.e- there's no such server or is not reacheble) Please post complete stack trace if you need further help on this.

[]s
Maurizio Villa
Greenhorn

Joined: Mar 24, 2009
Posts: 5
Hi All,
please, can you support me on the same issue?
I have a problem with a customer of mine.
We're facing BROKEN PIPE issue, on Weblogic ALSB 2.5.
This is teh error: java.net.SocketException: Broken pipe

This is the stack:

<Mar 17, 2009 9:48:03 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close:
java.net.SocketException: Write failed: Broken pipe
<Mar 17, 2009 9:48:41 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.net.SocketException: Write failed: Broken
pipe
<Mar 17, 2009 9:48:42 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close: java.net.SocketException: Write
failed: Broken pipe


.....

Following other details of the error:

<Mar 23, 2009 12:30:59 PM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpTransportServlet.service: java.net.SocketException: Write failed: Broken
pipe
java.net.SocketException: Write failed: Broken pipe
at jrockit.net.SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)
at jrockit.net.SocketNativeIO.socketWrite(Ljava.io.FileDescriptor;[BII)V(Unknown
Source)
at java.net.SocketOutputStream.socketWrite0(Ljava.io.FileDescriptor;[BII)V(SocketOu
tputStream.java:???)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Truncated. see log file for complete stacktrace
>
<Mar 23, 2009 12:31:00 PM CET> <Error> <HTTP> <BEA-101017>
<[weblogic.servlet.internal.WebAppServletContext@12affa8 - appName: 'XBus
Kernel', name: '/Portal/PX_CommonUtilityService', context-path:
'/Portal/PX_CommonUtilityService'] Root cause of
ServletException.
java.net.SocketException: Write failed: Broken pipe
at jrockit.net.SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)
at
jrockit.net.SocketNativeIO.socketWrite(Ljava.io.FileDescriptor;[BII)V(Unknown
Source)
at java.net.SocketOutputStream.socketWrite0(Ljava.io.FileDescriptor;[BII)V(SocketOu
tputStream.java:???)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Truncated. see log file for complete stacktrace


The issue is not dependent to the service in use.
The problem has a great priority due to the fact is in production env. and is causing loss of service.

Thanks in advance to everyone.
Maury
Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977

<Mar 17, 2009 9:48:41 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.net.SocketException: Write failed: Broken
pipe


Hi,

My first impression on this error is problably:

1) The client access the application
2) Some resource(DB, JCA connection, etc) is slow or some synchronized block in application is overloading the server and it takes too long to respond.
3) The client closes browser before receive the response and maybe try to start all over...
4) When the server tries to get back to client it's gone for that http cycle and that causes the Broken pipe... HttpInboundMessageContext.close....


My suggestion is for you to take some thread dumps when you realize the server is becaming overloaded and than you can find the problem(external resource or application dead lock).

[]s

P.S - To take a thread dump in Unix environments find the jvm process for a instance than use "kill -3 PID"




>
Maurizio Villa
Greenhorn

Joined: Mar 24, 2009
Posts: 5
Thanks Marcos,
please look this attachments, we're encountering queuing in thread pool unjustified.
have a look to this graph.

any hints?
regards.



[Thumbnail for Graph_Queue_Lenght.JPG]

Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977
That's it, you need few thread dumps from server in the moment the Queue and Hogging threads are happening.
In an interval from 13:27 to 13:34 minutes(or in a moment like this) you take some thread dumps, my sugestion is 2 or 3 per minute and than with this thread dump in hands you'll be able to find out exactly what's causing the queue and Hogging threads in your app.
Also problably you have a high number configured to consider threads as stuck(problably the default 600s - 10min) so you don't have stuck threads..

Take the thread dumps and you'll find out what's causing this problem(external resources or sync block code).
Are you using Jrockit JVM?

[]s
Maurizio Villa
Greenhorn

Joined: Mar 24, 2009
Posts: 5
Hi Marcos,
yes, JVM is JRockit -> Java Version: 1.5.0_06
The problem to get a full thread dump is to be so fast to intervene exactly when isssue is happening.

We also suspect some limit on DB, perhaps waits or something else.
I dont' know if it is a reasonable feeling.

Thanks for any kind of hints for addressing this issue.
Regards.
Maury
Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977
Well,

You don't need to do gess it, you can configure a notification to take diagnostic images(TDumps included), send snmp traps to notify someone who could run a script to take dumps, or automate TDumps in some flag situation, it's all about configuring or writting some code. you may also use WLST to take these dumps in some specific situation.

Take a look at the following link to read about notifications, I also have some WLST scripts who take dumps in some situation, if you wanna go this way let me know and I'll post some examples.

http://edocs.bea.com/wls/docs103/wldf_configuring/config_notifications.html

This is latest release but It's been around since 9.2 as far as I can remember.

[]s
Maurizio Villa
Greenhorn

Joined: Mar 24, 2009
Posts: 5
Hi M.,
thanks for suggestion, i really appreciate if you may share some of your WLST scripts, or send them to me via emai at
this address: mauriziovilla@tin.it so that i could perform some tests.
thanks in advance, i'll let you know.
regards.
Maurizio
Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977
Hi,

follows a sample script you can use as starting point, explanations follows. Unfotunatelly my primary language is Portuguese and you'll find comments on this language on this script.




First part of this script is used to load a properties file(follows) where you configure servers you wanto to monitor and load wladmin url, pwd, etc...

The function is the one who does the job.. the fragment:


Test how many hogging threads the server has and take a thread dump write to the file with pattern ThreadDump_SERVERNAME if the conditions is met.....

In your case I would modify this script to have a fuction wich would take for example one tdump each 20sec for a few cycles in case the hogging threads are > than certain number..

I would also modify it to keep conected to a desired server and each 1 min do a check on it..

This would do the job.

Follows the sample proerties:




The final consideration is the weakness of this approach which is related to servers being too overloaded in such case maybe you won't be able to take thread dumps or connect to the server(doesn't looks like your case and is a rare situation anyway).

[]s


Maurizio Villa
Greenhorn

Joined: Mar 24, 2009
Posts: 5
Thanks a lot.

I'll try them asap.

thanks for the moment.

As per your update thereupon the issue of broken pipe.
We saw even if we have stand by thread, when bea consumes all idle threads it starts to enque requests without using alla vailable stand by threads.
have you a concrete idea for this behaviour and the difference between idle and stand by status?
thanks.
regards
Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977
That's really strange to me! Can't remember from top of my mind what's the difference but as far as I remember Stand By threads are suppose to get picked to serve client requests.. really confused by this information.. Never seen this behavior.. I'll take a look and let you know if I find out something about this. It would be nice If you share your progress on that as well..

regards,
Marcos Maia.

Maurizio Villa wrote:Thanks a lot.

I'll try them asap.

thanks for the moment.

As per your update thereupon the issue of broken pipe.
We saw even if we have stand by thread, when bea consumes all idle threads it starts to enque requests without using alla vailable stand by threads.
have you a concrete idea for this behaviour and the difference between idle and stand by status?
thanks.
regards
Pavan BhavaniShekhar
Greenhorn

Joined: Nov 16, 2006
Posts: 9
Dear friends

To giving more flexibility to understand the ThreadPool Monitoring script using WLST when we use default Self-tune Work Manager.
I had renovated the script with English strings, Comments

Check it out!!

http://wlstbyexamples.blogspot.com/2009/06/self-tuned-thread-pool-count.html
suggest your needs and comments on the script.


Thanks & regards<br />Pavan BS Devarakonda<br />WebLogic Admin<br />Hyderabadi
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Issue with WeblogicServer
 
Similar Threads
Best villain character
MDB question.
JSP Forward
send response to calling servlet
ClassCastException with JMS.