aspose file tools*
The moose likes BEA/Weblogic and the fly likes WEBLOGIC GC: Getting BEA-000209 errors and web servers shuts down  Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Products » BEA/Weblogic
Bookmark "WEBLOGIC GC: Getting BEA-000209 errors and web servers shuts down  " Watch "WEBLOGIC GC: Getting BEA-000209 errors and web servers shuts down  " New topic
Author

WEBLOGIC GC: Getting BEA-000209 errors and web servers shuts down

Thaya Thava
Ranch Hand

Joined: Sep 17, 2008
Posts: 38
Hi All.
I have an issue with weblogic GC, which is not happening frequently where its all the way up at 90% and releasing a few % and then finally
the WLS crashes with 0% free. that is out of memory error.

on AIX 5.3 and running Weblogic 81 sp6, jre142_SR7 and i did ask them to turn on verbose gc and this is part of the stderr.log file:

<AF435: Allocation Failure. need 65552 bytes, 1414 ms since last AF>
<AF435: managing allocation failure, action=2 (979416/1073740288)>
<GC(756): freeing class sun.reflect.GeneratedSerializationConstructorAccessor467(9000c730)>
<GC(756): unloaded and freed 1 class>
<GC(756): GC cycle started Mon Mar 9 11:35:57 2009
><GC(756): freed 10035272 bytes, 1% free (11014688/1073740288), in 200 ms>
<GC(756): mark: 186 ms, sweep: 14 ms, compact: 0 ms>
<GC(756): refs: soft 0 (age >= 32), weak 0, final 136, phantom 0>
<AF435: managing allocation failure, action=3 (11014688/1073740288)>
<AF435: managing allocation failure, action=4 (11014688/1073740288)>
<AF435: clearing all remaining soft refs>
<GC(757): GC cycle started Mon Mar 9 11:35:58 2009
><GC(757): freed 80944 bytes, 1% free (11095632/1073740288), in 326 ms>
<GC(757): mark: 233 ms, sweep: 13 ms, compact: 80 ms>
<GC(757): refs: soft 24 (age >= 32), weak 0, final 1, phantom 0>
<GC(757): moved 64118 objects, 20352128 bytes, IC reason=5>
<AF435: completed in 548 ms>

<AF436: Allocation Failure. need 65552 bytes, 554 ms since last AF>
<AF436: managing allocation failure, action=2 (6607352/1073740288)>
<GC(758): freeing class sun.reflect.GeneratedSerializationConstructorAccessor468(9002b9f0)>
<GC(758): unloaded and freed 1 class>
<GC(758): GC cycle started Mon Mar 9 11:35:59 2009
><GC(758): freed 4765520 bytes, 1% free (11372872/1073740288), in 377 ms>
<GC(758): mark: 256 ms, sweep: 17 ms, compact: 104 ms>
<GC(758): refs: soft 0 (age >= 32), weak 0, final 74, phantom 0>
<GC(758): moved 81392 objects, 24747728 bytes, IC reason=5>
<AF436: managing allocation failure, action=3 (11372872/1073740288)>
<AF436: managing allocation failure, action=4 (11372872/1073740288)>
<AF436: clearing all remaining soft refs>
<GC(759): GC cycle started Mon Mar 9 11:35:59 2009
><GC(759): freed 63816 bytes, 1% free (11436688/1073740288), in 324 ms>
<GC(759): mark: 234 ms, sweep: 13 ms, compact: 77 ms>
<GC(759): refs: soft 12 (age >= 32), weak 0, final 1, phantom 0>
<GC(759): moved 71287 objects, 21891112 bytes, IC reason=5>
<AF436: completed in 704 ms>

<AF437: Allocation Failure. need 65552 bytes, 2924 ms since last AF>
<AF437: managing allocation failure, action=2 (1295984/1073740288)>
<GC(760): freeing class sun.reflect.GeneratedSerializationConstructorAccessor469(9000c730)>
<GC(760): unloaded and freed 1 class>
<GC(760): GC cycle started Mon Mar 9 11:36:02 2009
><GC(760): freed 9244952 bytes, 0% free (10540936/1073740288), in 337 ms>
<GC(760): mark: 256 ms, sweep: 13 ms, compact: 68 ms>
<GC(760): refs: soft 0 (age >= 32), weak 0, final 171, phantom 0>
<GC(760): moved 47010 objects, 20641336 bytes, IC reason=5>
<AF437: managing allocation failure, action=3 (10540936/1073740288)>
<AF437: managing allocation failure, action=4 (10540936/1073740288)>
<AF437: clearing all remaining soft refs>
<GC(761): GC cycle started Mon Mar 9 11:36:03 2009
><GC(761): freed 100328 bytes, 0% free (10641264/1073740288), in 329 ms>
<GC(761): mark: 230 ms, sweep: 14 ms, compact: 85 ms>
<GC(761): refs: soft 24 (age >= 32), weak 0, final 1, phantom 0>
<GC(761): moved 124854 objects, 44831488 bytes, IC reason=5>
<AF437: completed in 669 ms>

<AF438: Allocation Failure. need 65552 bytes, 6745 ms since last AF>
<AF438: managing allocation failure, action=2 (695880/1073740288)>
<GC(762): freeing class sun.reflect.GeneratedSerializationConstructorAccessor470(9002b9f0)>
<GC(762): unloaded and freed 1 class>
<GC(762): GC cycle started Mon Mar 9 11:36:10 2009
><GC(762): freed 11778704 bytes, 1% free (12474584/1073740288), in 367 ms>
<GC(762): mark: 262 ms, sweep: 13 ms, compact: 92 ms>
<GC(762): refs: soft 0 (age >= 32), weak 0, final 273, phantom 0>
<GC(762): moved 134987 objects, 57955208 bytes, IC reason=5>
<AF438: managing allocation failure, action=3 (12474584/1073740288)>
<AF438: managing allocation failure, action=4 (12474584/1073740288)>
<AF438: clearing all remaining soft refs>
<GC(763): GC cycle started Mon Mar 9 11:36:10 2009
><GC(763): freed 105288 bytes, 1% free (12579872/1073740288), in 319 ms>
<GC(763): mark: 232 ms, sweep: 14 ms, compact: 73 ms>
<GC(763): refs: soft 24 (age >= 32), weak 0, final 1, phantom 0>
<GC(763): moved 56499 objects, 26667024 bytes, IC reason=5>
<AF438: completed in 689 ms>
<GC(764): freeing class sun.reflect.GeneratedSerializationConstructorAccessor471(9000c730)>
<GC(764): unloaded and freed 1 class>

<GC(764): GC cycle started Mon Mar 9 11:36:17 2009
><GC(764): freed 5057904 bytes, 1% free (12374336/1073740288), in 1452 ms>
<GC(764): mark: 254 ms, sweep: 15 ms, compact: 1183 ms>
<GC(764): refs: soft 0 (age >= 32), weak 0, final 159, phantom 0>
<GC(764): moved 2739808 objects, 932954672 bytes, reason=4, used 6952 more bytes>

<AF439: Allocation Failure. need 528 bytes, 12003 ms since last AF>
<AF439: managing allocation failure, action=2 (0/1073740288)>
<GC(765): GC cycle started Mon Mar 9 11:36:22 2009
><GC(765): freed 10961288 bytes, 1% free (10961288/1073740288), in 203 ms>
<GC(765): mark: 190 ms, sweep: 13 ms, compact: 0 ms>
<GC(765): refs: soft 0 (age >= 32), weak 0, final 217, phantom 0>
<AF439: managing allocation failure, action=3 (10961288/1073740288)>
<AF439: managing allocation failure, action=4 (10961288/1073740288)>
<AF439: clearing all remaining soft refs>
<GC(766): GC cycle started Mon Mar 9 11:36:23 2009
><GC(766): freed 118936 bytes, 1% free (11080224/1073740288), in 291 ms>
<GC(766): mark: 228 ms, sweep: 14 ms, compact: 49 ms>
<GC(766): refs: soft 27 (age >= 32), weak 0, final 1, phantom 0>
<GC(766): moved 21238 objects, 8014744 bytes, IC reason=5>
<AF439: completed in 506 ms>

<AF440: Allocation Failure. need 65552 bytes, 1227 ms since last AF>
<AF440: managing allocation failure, action=2 (3040440/1073740288)>
<GC(767): freeing class sun.reflect.GeneratedSerializationConstructorAccessor472(9002b9f0)>
<GC(767): unloaded and freed 1 class>
<GC(767): GC cycle started Mon Mar 9 11:36:24 2009
><GC(767): freed 7486104 bytes, 0% free (10526544/1073740288), in 298 ms>
<GC(767): mark: 244 ms, sweep: 13 ms, compact: 41 ms>
<GC(767): refs: soft 0 (age >= 32), weak 0, final 104, phantom 0>
<GC(767): moved 1610 objects, 1387256 bytes, IC reason=5>
<AF440: managing allocation failure, action=3 (10526544/1073740288)>
<AF440: managing allocation failure, action=4 (10526544/1073740288)>
<AF440: clearing all remaining soft refs>
<GC(768): GC cycle started Mon Mar 9 11:36:24 2009
><GC(768): freed 95128 bytes, 0% free (10621672/1073740288), in 291 ms>
<GC(768): mark: 228 ms, sweep: 13 ms, compact: 50 ms>
<GC(768): refs: soft 22 (age >= 32), weak 0, final 1, phantom 0>
<GC(768): moved 11896 objects, 6294960 bytes, IC reason=5>
<AF440: completed in 592 ms>

Did set their ulimit (setenv.sh) to be :
#---------------------------------------------

1. Skipping JDK Check for current PeopleTools release.
2. At this time PeopleTools only requires a JDK. This may change.

#findJDK
#---------------------------------------------

1. Grab some file descriptors.

if ; then
if ; then
maxfiles=`ulimit -n`
else
maxfiles=`ulimit -H -n`
fi
if ; then
maxfiles=1024
fi
if ; then
maxfiles=1024
fi
ulimit -n 8192
fi

oh, one more thing their setting is, right now the jave heap is set to 1gb(min and max), used to be 2gb(min and max) and still weblogic crashed :

JAVA_OPTIONS_AIX="-Xms1024m -Xmx1024m"

env setup:
big-ip load balancer , 2 proxy servers, 5 web servers, 2 content servers and 5 application server domains

initial error :

PIA_weblogic.log:####<Feb 27, 2009 10:50:24 AM CST> <Emergency> <WebLogicServer> <ma1ua275> <PIA> <ListenThread.Default> <<WLS Kernel>> <BEA-000209> <ListenThread.run() failed: java.lang.OutOfMemoryError
PIA_weblogic.log:java.lang.OutOfMemoryError

min and max heap size is 1024mb and this server has 16Gb of RAM.

Any thoughts...>
Marcos Maia
Ranch Hand

Joined: Jan 06, 2001
Posts: 977
Hi,

Under AIX you're probably using IBM jvm, so I'd suggest you to try to get help in websphere or IBM forums. About this kind of problem it's usually a memory leak in the app running under weblogic wich cuses it. You'll need a profile tool to monitor this IBM jvm you're using to find out where and when is the memory leak happening.

[]s
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: WEBLOGIC GC: Getting BEA-000209 errors and web servers shuts down