• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Issue with WeblogicServer

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All

I am getting below mantioned error while stoping the webserver

Failed to connect to t3://MYSERVER:8098: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination

Please help me out.....
 
Ranch Hand
Posts: 32
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We get this when there are any Connection Reset Errors. Check your connection pool and data source.
 
Bartender
Posts: 2856
10
Firefox Browser Fedora Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
"Amit Kumar Hcl " please check your private messages for an important administrative matter. You can check them by clicking the My Private Messages link above.
 
Ranch Hand
Posts: 977
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Piyush Kumar Sharma wrote:Hi All

I am getting below mantioned error while stoping the webserver

Failed to connect to t3://MYSERVER:8098: Destination unreachable; nested exception is:
java.net.SocketException: Broken pipe; No available router to destination

Please help me out.....



Usually that's exactly what means it can't resolve MYSERVER:8098 as a weblogic server instance(i.e- there's no such server or is not reacheble) Please post complete stack trace if you need further help on this.

[]s
 
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,
please, can you support me on the same issue?
I have a problem with a customer of mine.
We're facing BROKEN PIPE issue, on Weblogic ALSB 2.5.
This is teh error: java.net.SocketException: Broken pipe

This is the stack:

<Mar 17, 2009 9:48:03 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close:
java.net.SocketException: Write failed: Broken pipe
<Mar 17, 2009 9:48:41 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.net.SocketException: Write failed: Broken
pipe
<Mar 17, 2009 9:48:42 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in HttpInboundMessageContext.close: java.net.SocketException: Write
failed: Broken pipe


.....

Following other details of the error:

<Mar 23, 2009 12:30:59 PM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpTransportServlet.service: java.net.SocketException: Write failed: Broken
pipe
java.net.SocketException: Write failed: Broken pipe
at jrockit.net.SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)
at jrockit.net.SocketNativeIO.socketWrite(Ljava.io.FileDescriptor;[BII)V(Unknown
Source)
at java.net.SocketOutputStream.socketWrite0(Ljava.io.FileDescriptor;[BII)V(SocketOu
tputStream.java:???)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Truncated. see log file for complete stacktrace
>
<Mar 23, 2009 12:31:00 PM CET> <Error> <HTTP> <BEA-101017>
<[weblogic.servlet.internal.WebAppServletContext@12affa8 - appName: 'XBus
Kernel', name: '/Portal/PX_CommonUtilityService', context-path:
'/Portal/PX_CommonUtilityService'] Root cause of
ServletException.
java.net.SocketException: Write failed: Broken pipe
at jrockit.net.SocketNativeIO.writeBytesPinned(I[BII)V(Native Method)
at
jrockit.net.SocketNativeIO.socketWrite(Ljava.io.FileDescriptor;[BII)V(Unknown
Source)
at java.net.SocketOutputStream.socketWrite0(Ljava.io.FileDescriptor;[BII)V(SocketOu
tputStream.java:???)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
Truncated. see log file for complete stacktrace


The issue is not dependent to the service in use.
The problem has a great priority due to the fact is in production env. and is causing loss of service.

Thanks in advance to everyone.
Maury
 
Marcos Maia
Ranch Hand
Posts: 977
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator


<Mar 17, 2009 9:48:41 AM CET> <Error> <WliSbTransports> <BEA-381304> <Exception in
HttpInboundMessageContext.close: java.net.SocketException: Write failed: Broken
pipe



Hi,

My first impression on this error is problably:

1) The client access the application
2) Some resource(DB, JCA connection, etc) is slow or some synchronized block in application is overloading the server and it takes too long to respond.
3) The client closes browser before receive the response and maybe try to start all over...
4) When the server tries to get back to client it's gone for that http cycle and that causes the Broken pipe... HttpInboundMessageContext.close....


My suggestion is for you to take some thread dumps when you realize the server is becaming overloaded and than you can find the problem(external resource or application dead lock).

[]s

P.S - To take a thread dump in Unix environments find the jvm process for a instance than use "kill -3 PID"




>
 
Maurizio Villa
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Marcos,
please look this attachments, we're encountering queuing in thread pool unjustified.
have a look to this graph.

any hints?
regards.
Graph_Queue_Lenght.JPG
[Thumbnail for Graph_Queue_Lenght.JPG]
Thread pool status
 
Marcos Maia
Ranch Hand
Posts: 977
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
That's it, you need few thread dumps from server in the moment the Queue and Hogging threads are happening.
In an interval from 13:27 to 13:34 minutes(or in a moment like this) you take some thread dumps, my sugestion is 2 or 3 per minute and than with this thread dump in hands you'll be able to find out exactly what's causing the queue and Hogging threads in your app.
Also problably you have a high number configured to consider threads as stuck(problably the default 600s - 10min) so you don't have stuck threads..

Take the thread dumps and you'll find out what's causing this problem(external resources or sync block code).
Are you using Jrockit JVM?

[]s
 
Maurizio Villa
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Marcos,
yes, JVM is JRockit -> Java Version: 1.5.0_06
The problem to get a full thread dump is to be so fast to intervene exactly when isssue is happening.

We also suspect some limit on DB, perhaps waits or something else.
I dont' know if it is a reasonable feeling.

Thanks for any kind of hints for addressing this issue.
Regards.
Maury
 
Marcos Maia
Ranch Hand
Posts: 977
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well,

You don't need to do gess it, you can configure a notification to take diagnostic images(TDumps included), send snmp traps to notify someone who could run a script to take dumps, or automate TDumps in some flag situation, it's all about configuring or writting some code. you may also use WLST to take these dumps in some specific situation.

Take a look at the following link to read about notifications, I also have some WLST scripts who take dumps in some situation, if you wanna go this way let me know and I'll post some examples.

http://edocs.bea.com/wls/docs103/wldf_configuring/config_notifications.html

This is latest release but It's been around since 9.2 as far as I can remember.

[]s
 
Maurizio Villa
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi M.,
thanks for suggestion, i really appreciate if you may share some of your WLST scripts, or send them to me via emai at
this address: mauriziovilla@tin.it so that i could perform some tests.
thanks in advance, i'll let you know.
regards.
Maurizio
 
Marcos Maia
Ranch Hand
Posts: 977
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

follows a sample script you can use as starting point, explanations follows. Unfotunatelly my primary language is Portuguese and you'll find comments on this language on this script.




First part of this script is used to load a properties file(follows) where you configure servers you wanto to monitor and load wladmin url, pwd, etc...

The function is the one who does the job.. the fragment:


Test how many hogging threads the server has and take a thread dump write to the file with pattern ThreadDump_SERVERNAME if the conditions is met.....

In your case I would modify this script to have a fuction wich would take for example one tdump each 20sec for a few cycles in case the hogging threads are > than certain number..

I would also modify it to keep conected to a desired server and each 1 min do a check on it..

This would do the job.

Follows the sample proerties:




The final consideration is the weakness of this approach which is related to servers being too overloaded in such case maybe you won't be able to take thread dumps or connect to the server(doesn't looks like your case and is a rare situation anyway).

[]s


 
Maurizio Villa
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks a lot.

I'll try them asap.

thanks for the moment.

As per your update thereupon the issue of broken pipe.
We saw even if we have stand by thread, when bea consumes all idle threads it starts to enque requests without using alla vailable stand by threads.
have you a concrete idea for this behaviour and the difference between idle and stand by status?
thanks.
regards
 
Marcos Maia
Ranch Hand
Posts: 977
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
That's really strange to me! Can't remember from top of my mind what's the difference but as far as I remember Stand By threads are suppose to get picked to serve client requests.. really confused by this information.. Never seen this behavior.. I'll take a look and let you know if I find out something about this. It would be nice If you share your progress on that as well..

regards,
Marcos Maia.

Maurizio Villa wrote:Thanks a lot.

I'll try them asap.

thanks for the moment.

As per your update thereupon the issue of broken pipe.
We saw even if we have stand by thread, when bea consumes all idle threads it starts to enque requests without using alla vailable stand by threads.
have you a concrete idea for this behaviour and the difference between idle and stand by status?
thanks.
regards

 
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Dear friends

To giving more flexibility to understand the ThreadPool Monitoring script using WLST when we use default Self-tune Work Manager.
I had renovated the script with English strings, Comments

Check it out!!

http://wlstbyexamples.blogspot.com/2009/06/self-tuned-thread-pool-count.html
suggest your needs and comments on the script.
 
reply
    Bookmark Topic Watch Topic
  • New Topic