BEA Weblogic Server goes down

PTools:8.47.13 and application release 88sp1
WLS: 81 sp5
OS: windows server 2003

Having to reboot services for Peoplebooks almost every other day.

WL LOG:
####<Sep 9, 2008 10:31:45 AM CDT> <Info> <HTTP> <CHIPPSWEB01> <PSOL>
<ExecuteThread: '48' for queue: 'weblogic.kernel.Default'> <<anonymous>>
<> <BEA-101047>
<[ServletContext(id=6170099,name=PSOL,context-path=/PSOL)] init()
invoked>
####<Sep 9, 2008 4:56:15 PM CDT> <Error> <Socket> <CHIPPSWEB01> <PSOL>
<ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>>
<> <BEA-000421> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError.
java.lang.OutOfMemoryError
>
####<Sep 9, 2008 5:21:23 PM CDT> <Emergency> <WebLogicServer>
<CHIPPSWEB01> <PSOL> <ListenThread.Default> <<WLS Kernel>> <>
<BEA-000210> <The WebLogic Server is no longer listening for
connections.>
####<Sep 9, 2008 5:21:23 PM CDT> <Alert> <WebLogicServer> <CHIPPSWEB01>
<PSOL> <ListenThread.Default> <<WLS Kernel>> <> <BEA-000218> <Server
shutdown has been requested by <WLS Kernel>>
####<Sep 10, 2008 2:30:45 AM CDT> <Warning> <WebLogicServer>
<CHIPPSWEB01> <PSOL> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>>
<> <BEA-000335> <Percent of starting server free memory is now at "1%".
This is less than the configured threshold of "5%". We will try to
garbage collect, but you may consider rebooting.>

Is there a file where I need to increase page file or something where
this memory issue will not happen every other day

try changing memory arguments in the startWeblogic.cmd file :
set MEM_ARGS="-Xms:256m - Xmx:512m"

try changing memory arguments in the startWeblogic.cmd file :
set MEM_ARGS="-Xms:256m - Xmx:512m"

You mean to say the JAVA_OPTIONS in setenv.cmd file, think i have it upto 1024(min and max).
Do i need to add -XX254 to the end of the command?
think its something to do with GC, which is not happening...
thanks

Hi,

First you can try tunning your JVM. Are you using JRockit? Please specify the JVM you�re using.
You should also look for possible memory leaks in you application.
After posting JVM I will be able to give better help.

regards.

Hi Marcos.
JVM is set to:
SET JAVA_OPTIONS_WIN32=-server -Xms1024m -Xmx1024m -XX:MaxPermSize=256m

JRochit:

awt.toolkit = sun.awt.windows.WToolkit
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = ;D:/PSOFT/SNRHCPRD\webserv\snrhcprd\lib\ps_patch.jar;D:/PSOFT/SNRHCPRD\webserv\snrhcprd\lib\portlet-api-1.0.jar;D:/PSOFT/SNRHCPRD\webserv\snrhcprd\lib\pluto-1.0.1.jar;D:/PSOFT/SNRHCPRD\webserv\snrhcprd\applications\peoplesoft\PORTAL\WEB-INF\lib\saaj.jar;D:/PSOFT/SNRHCPRD\webserv\snrhcprd\applications\peoplesoft\PORTAL\WEB-INF\lib\xml-apis.jar;D:/PSOFT/SNRHCPRD\webserv\snrhcprd\applications\peoplesoft\PORTAL\WEB-INF\lib\xercesImpl.jar;d:\bea\WEBLOG~1\server\lib\Win_32_81SP5_32_supplement.jar;d:\b ea\jdk142_08\lib\tools.jar;d:\bea\WEBLOG~1\server\lib\weblogic_sp.jar;d:\bea\WEBLOG~1\server\lib\weblogic.jar
java.class.version = 48.0
java.endorsed.dirs = d:\bea\jdk142_08\jre\lib\endorsed
java.ext.dirs = d:\bea\jdk142_08\jre\lib\ext
java.home = d:\bea\jdk142_08\jre
java.io.tmpdir = C:\WINDOWS\TEMP\
java.library.path = d:\bea\WEBLOG~1\server\bin;.;C:\WINDOWS\system32;C:\WINDOWS;d:\bea\WEBLOG~1\server\bin;d:\bea\jdk142_08\bin;D:\PSOFT\SNRHCPRD\webserv\snrhcprd\applications\peoplesoft\PSOL\WEB-INF\lib\winx86;D:\PSOFT\SNRHCPRD\verity\winx86\_nti40\bin;C:\Program Files\Support Tools\;d:\bea\jdk142_08;d:\psoft\bin\client\winx86;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\HITACHI\DynamicLinkManager\bin;C:\Program Files\HITACHI\DynamicLinkManager\lib;C:\Program Files\Common Files\Hitachi;d:\bea\tuxedo8.1\bin;C:\Program Files\Microsoft SQL Server\80\Tools\BINN;C:\Program Files\System Center Operations Manager 2007\;D:\PSOFT\SNRHCPRD\webserv\snrhcprd\applications\peoplesoft\PSOL\WEB-INF\lib\winx86
java.naming.factory.initial = weblogic.jndi.WLInitialContextFactory
java.naming.factory.url.pkgs = weblogic.jndi.factories:weblogic.corba.j2ee.naming.url
java.protocol.handler.pkgs = weblogic.utils|weblogic.utils|weblogic.net
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version = 1.4.2_08-b03
java.security.policy = d:\bea\WEBLOG~1\server\lib\weblogic.policy
java.specification.name = Java Platform API Specification
java.specification.vendor = Sun Microsystems Inc.
java.specification.version = 1.4
java.util.logging.config.file = ./applications/peoplesoft/logging.properties
java.util.prefs.PreferencesFactory = java.util.prefs.WindowsPreferencesFactory
java.vendor = Sun Microsystems Inc.
java.vendor.url = http://java.sun.com/
java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
java.version = 1.4.2_08
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) Client VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = Sun Microsystems Inc.
java.vm.version = 1.4.2_08-b03
javax.rmi.CORBA.PortableRemoteObjectClass = weblogic.iiop.PortableRemoteObjectDelegateImpl
javax.rmi.CORBA.UtilClass = weblogic.iiop.UtilDelegateImpl
javax.xml.rpc.ServiceFactory = weblogic.webservice.core.rpc.ServiceFactoryImpl
javax.xml.soap.MessageFactory = weblogic.webservice.core.soap.MessageFactoryImpl
jmx.implementation.name = JMX RI
jmx.implementation.vendor = Sun Microsystems
jmx.implementation.version = 1.0
jmx.specification.name = Java Management Extensions
jmx.specification.vendor = Sun Microsystems
jmx.specification.version = 1.0 Final Release
org.apache.commons.logging.Log = org.apache.commons.logging.impl.Jdk14Logger
org.omg.CORBA.ORBClass = weblogic.corba.orb.ORB
org.omg.CORBA.ORBSingletonClass = weblogic.corba.orb.ORB
org.xml.sax.driver = weblogic.apache.xerces.parsers.SAXParser
org.xml.sax.parser = weblogic.xml.jaxp.RegistryParser
os.arch = x86
os.name = Windows 2003
os.version = 5.2
path.separator = ;
ps_home = D:/PSOFT/SNRHCPRD
ps_vault = D:/PSOFT/SNRHCPRD\secvault\psvault
ssl.debug = false
sun.arch.data.model = 32
sun.boot.class.path = d:\bea\jdk142_08\jre\lib\rt.jar;d:\bea\jdk142_08\jre\lib\i18n.jar;d:\bea\jdk142_08\jre\lib\sunrsasign.jar;d:\bea\jdk142_08\jre\lib\jsse.jar;d:\bea\jdk142_08\jre\lib\jce.jar;d:\bea\jdk142_08\jre\lib\charsets.jar;d:\bea\jdk142_08\jre\classes
sun.boot.library.path = d:\bea\jdk142_08\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = pentium i486 i386
sun.io.unicode.encoding = UnicodeLittle
sun.os.patch.level = Service Pack 1
user.country = US
user.dir = D:\PSOFT\SNRHCPRD\webserv\snrhcprd
user.home = C:\Documents and Settings\Default User
user.language = en
user.name = SYSTEM
user.timezone = America/Chicago
vde.home = .\PSOL\ldap
weblogic.Name = PSOL
weblogic.ProductionModeEnabled = true
weblogic.management.server = HTTP://00.000.000.000:0000
weblogic.management.startmode = WinSvc
>

How do i see memeory leak in the application?
this is the error that im still seeing on WL log:

####<Sep 19, 2008 5:34:42 AM CDT> <Error> <Kernel> <CHIPPSWEB01> <PSOL> <ExecuteThread: '3' for queue: 'weblogic.kernel.System'> <<WLS Kernel>> <> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError.
java.lang.OutOfMemoryError
>
####<Sep 19, 2008 5:36:40 AM CDT> <Emergency> <WebLogicServer> <CHIPPSWEB01> <PSOL> <ListenThread.Default> <<WLS Kernel>> <> <BEA-000209> <ListenThread.run() failed: java.lang.OutOfMemoryError
java.lang.OutOfMemoryError

regards/t

Hi,

first thing is to change your environment to use JRockit, as I can see from you list you're using Sun jdk:

java.home = d:\bea\jdk142_08\jre (from your input)

Take a look at:

http://e-docs.bea.com/platform/docs81/install/postins.html#1083410

to check exactly wich version you're using, if not using JRockit(as I believe)

first try changing it to use jrockit that problably exists under

d:\bea\jrockig142_08

or in some directory with similar name.

To do this edit the setDomainEnv.cmd script under %DOMAIN_HOME%\bin and add the entry:

set JAVA_VENDOR=BEA

just before the line:

if "%JAVA_VENDOR%"=="BEA" (

Restart your environment and try it with default JRockit configuration for a while, if the problems still persists than you can try changing the garbage collector algoritms depending on your application needs.

Take a good look at:

http://edocs.bea.com/jrockit/docs142/tuning/index.html
(problably this is the jrockit version you're using, but please recheck it to look at the correct documentation.

There you'll also find information on how to look for memory leaks in your application if this is the case.

Please let us know about progress on this, reporting your progress here.

regards.

Hi Macros,

I too have faced the similar issue today.

Suddenly, i saw this error msg in both the managed servers running in production mode, which made the thread count too low and later resulted me to restart the servers.

####<04.jan.2012 kl 15.49 CET> <Warning> <WebLogicServer> <HIPD144APP20> <intHipd144app20> <JMAPI event thread> <<WLS Kernel>> <> <BEA-000335> <Percent of starting server free memory is now at "4%". This is less than the configured threshold of "5%". We will try to garbage collect, but you may consider rebooting.>

awt.toolkit = sun.awt.windows.WToolkit
bea.home = C:\bea_8_1_4
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = C:\bea_8_1_4\weblogic81\server\lib\cr210310_81sp4.jar;C:\bea_8_1_4\weblogic81\server\lib\weblogic_knex_patch.jar;C:\bea_8_1_4\weblogic81\common\lib\log4j.jar;C:\bea_8_1_4\weblogic81\server\lib\debugging.jar;C:\bea_8_1_4\weblogic81\server\lib\knex.jar;C:\bea_8_1_4\weblogic81\javelin\lib\javelin.jar;C:\bea_8_1_4\weblogic81\server\lib\wlw-lang.jar;C:\bea_8_1_4\jrockit81sp4_142_05\lib\tools.jar;C:\bea_8_1_4\weblogic81\server\lib\weblogic_sp.jar;C:\bea_8_1_4\weblogic81\server\lib\weblogic.jar;C:\bea_8_1_4\weblogic81\server\lib\ant\ant.jar;C:\bea_8_1_4\jrockit81sp4_142_05\jre\lib\rt.jar;C:\bea_8_1_4\weblogic81\common\eval\pointbase\lib\pbclient44.jar;C:\bea_8_1_4\weblogic81\server\lib\webserviceclient.jar;C:\bea_8_1_4\weblogic81\server\lib\webserviceclient+ssl.jar;C:\bea_8_1_4\weblogic81\server\lib\xbean.jar;C:\bea_8_1_4\weblogic81\server\lib\wlxbean.jar;C:\bea_8_1_4\weblogic81\server\lib\xqrl.jar;C:\bea_8_1_4\weblogic81\server\lib\netui\netui-compiler.jar;C:\bea_8_1_4\weblogic81\server\lib\wli.jar;C:\bea_8_1_4\weblogic81\server\lib\fop.jar;C:\bea_8_1_4\weblogic81\integration\adapters\sample\lib\sample-eis.jar;C:\bea_8_1_4\weblogic81\p13n\lib\p13n_system.jar
java.class.version = 48.0
java.endorsed.dirs = C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\endorsed
java.ext.dirs = C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\ext
java.home = C:\bea_8_1_4\jrockit81sp4_142_15\jre
java.io.tmpdir = C:\WINDOWS\TEMP\
java.library.path = C:\bea_8_1_4\jrockit81sp4_142_15\bin;.;C:\WINDOWS\system32;C:\WINDOWS;C:\BEA_8_~1\WEBLOG~1\server\bin;C:\bea_8_1_4\jdk142_05\bin;C:\PROGRA~1\CA\SC\\ETPKI\lib;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBEM;C:\Program Files\CA\Dcs\DMScripting\;C:\Program Files\CA\DCS\CAWIN\;C:\Program Files\Windows Resource Kits\Tools\;C:\Program Files\Support Tools\;C:\Program Files\Microsoft ADS\bin;C:\Program Files\Microsoft ADS\tools;C:\NSM\help;C:\Program Files\CA\SharedComponents\ScanEngine;C:\Program Files\FalconStor\IPStor\Dynapath\
java.naming.factory.initial = weblogic.jndi.WLInitialContextFactory
java.naming.factory.url.pkgs = weblogic.jndi.factories:weblogic.corba.j2ee.naming.url
java.protocol.handler.pkgs = weblogic.utils|weblogic.utils|weblogic.net|com.bea.b2b.transport
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version = 1.4.2_15-b02
java.security.policy = =C:\bea_8_1_4\weblogic81\server\lib\weblogic.policy
java.specification.name = Java Platform API Specification
java.specification.vendor = Sun Microsystems Inc.
java.specification.version = 1.4
java.util.prefs.PreferencesFactory = java.util.prefs.WindowsPreferencesFactory
java.vendor = BEA Systems, Inc.
java.vendor.url = http://www.bea.com/
java.vendor.url.bug = http://support.bea.com
java.version = 1.4.2_15
java.vm.info = compiled mode
java.vm.name = BEA JRockit(R)
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = BEA Systems, Inc.
java.vm.vendor.url = http://www.bea.com/
java.vm.vendor.url.bug = http://support.bea.com
java.vm.version = R27.4.0-90_CR352234-91983-1.4.2_15-20071115-1548-windows-ia32
javax.rmi.CORBA.PortableRemoteObjectClass = weblogic.iiop.PortableRemoteObjectDelegateImpl
javax.rmi.CORBA.UtilClass = weblogic.iiop.UtilDelegateImpl
javax.xml.rpc.ServiceFactory = weblogic.webservice.core.rpc.ServiceFactoryImpl
javax.xml.soap.MessageFactory = weblogic.webservice.core.soap.MessageFactoryImpl
jmx.implementation.name = JMX RI
jmx.implementation.vendor = Sun Microsystems
jmx.implementation.version = 1.0
jmx.specification.name = Java Management Extensions
jmx.specification.vendor = Sun Microsystems
jmx.specification.version = 1.0 Final Release
log4j.configuration = file

:\UAN_Home\Log4jConfig\LogCfg4hipad20.xml
org.omg.CORBA.ORBClass = weblogic.corba.orb.ORB
org.omg.CORBA.ORBSingletonClass = weblogic.corba.orb.ORB
org.xml.sax.driver = weblogic.apache.xerces.parsers.SAXParser
org.xml.sax.parser = weblogic.xml.jaxp.RegistryParser
os.arch = x86
os.name = Windows 2003
os.version = 5.2
path.separator = ;
sun.arch.data.model = 32
sun.boot.class.path = C:\bea_8_1_4\jrockit81sp4_142_15\jre\bin\jrockit\jrockit1.4.2.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\bin\jrockit\jmapi.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\bin\jrockit\jmxmapi.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\bin\jrockit\rmp.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\bin\jrockit\latency.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\rt.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\i18n.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\sunrsasign.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\jsse.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\jce.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\lib\charsets.jar;C:\bea_8_1_4\jrockit81sp4_142_15\jre\classes
sun.boot.library.path = C:\bea_8_1_4\jrockit81sp4_142_15\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = pentium i486 i386
sun.io.unicode.encoding = UnicodeLittle
sun.os.patch.level = Service Pack 1
user.country = NO
user.dir = D:\bea_files\va_intdomain
user.home = C:\WINDOWS\system32\config\systemprofile
user.language = no
user.name = SYSTEM
user.timezone = Europe/Paris
vde.home = D:\bea_files\va_intdomain\intHipd144app20\ldap
weblogic.Name = intHipd144app20
weblogic.ReverseDNSAllowed = false
weblogic.RootDirectory = D:\bea_files\va_intdomain
weblogic.management.server = http://172.19.3.171:7001
weblogic.nodemanager.ServerStartTime = 1325693281540
weblogic.security.SSL.ignoreHostnameVerification = true
weblogic.security.TrustKeyStore = DemoTrust
weblogic.system.BootIdentityFile = D:\bea_files\nodemanager\NodeManagerLogs\NodeManagerInternal\bootFile_va_intdomain_intHipd144app20
weblogic.system.NodeManagerAddress = null::5555
weblogic.system.NodeManagerBoot = true