Win a copy of Clojure in Action this week in the Clojure forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

WEBLOGIC GC: Getting BEA-000209 errors and web servers shuts down

 
Thaya Thava
Ranch Hand
Posts: 38
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All.
I have an issue with weblogic GC, which is not happening frequently where its all the way up at 90% and releasing a few % and then finally
the WLS crashes with 0% free. that is out of memory error.

on AIX 5.3 and running Weblogic 81 sp6, jre142_SR7 and i did ask them to turn on verbose gc and this is part of the stderr.log file:

<AF435: Allocation Failure. need 65552 bytes, 1414 ms since last AF>
<AF435: managing allocation failure, action=2 (979416/1073740288)>
<GC(756): freeing class sun.reflect.GeneratedSerializationConstructorAccessor467(9000c730)>
<GC(756): unloaded and freed 1 class>
<GC(756): GC cycle started Mon Mar 9 11:35:57 2009
><GC(756): freed 10035272 bytes, 1% free (11014688/1073740288), in 200 ms>
<GC(756): mark: 186 ms, sweep: 14 ms, compact: 0 ms>
<GC(756): refs: soft 0 (age >= 32), weak 0, final 136, phantom 0>
<AF435: managing allocation failure, action=3 (11014688/1073740288)>
<AF435: managing allocation failure, action=4 (11014688/1073740288)>
<AF435: clearing all remaining soft refs>
<GC(757): GC cycle started Mon Mar 9 11:35:58 2009
><GC(757): freed 80944 bytes, 1% free (11095632/1073740288), in 326 ms>
<GC(757): mark: 233 ms, sweep: 13 ms, compact: 80 ms>
<GC(757): refs: soft 24 (age >= 32), weak 0, final 1, phantom 0>
<GC(757): moved 64118 objects, 20352128 bytes, IC reason=5>
<AF435: completed in 548 ms>

<AF436: Allocation Failure. need 65552 bytes, 554 ms since last AF>
<AF436: managing allocation failure, action=2 (6607352/1073740288)>
<GC(758): freeing class sun.reflect.GeneratedSerializationConstructorAccessor468(9002b9f0)>
<GC(758): unloaded and freed 1 class>
<GC(758): GC cycle started Mon Mar 9 11:35:59 2009
><GC(758): freed 4765520 bytes, 1% free (11372872/1073740288), in 377 ms>
<GC(758): mark: 256 ms, sweep: 17 ms, compact: 104 ms>
<GC(758): refs: soft 0 (age >= 32), weak 0, final 74, phantom 0>
<GC(758): moved 81392 objects, 24747728 bytes, IC reason=5>
<AF436: managing allocation failure, action=3 (11372872/1073740288)>
<AF436: managing allocation failure, action=4 (11372872/1073740288)>
<AF436: clearing all remaining soft refs>
<GC(759): GC cycle started Mon Mar 9 11:35:59 2009
><GC(759): freed 63816 bytes, 1% free (11436688/1073740288), in 324 ms>
<GC(759): mark: 234 ms, sweep: 13 ms, compact: 77 ms>
<GC(759): refs: soft 12 (age >= 32), weak 0, final 1, phantom 0>
<GC(759): moved 71287 objects, 21891112 bytes, IC reason=5>
<AF436: completed in 704 ms>

<AF437: Allocation Failure. need 65552 bytes, 2924 ms since last AF>
<AF437: managing allocation failure, action=2 (1295984/1073740288)>
<GC(760): freeing class sun.reflect.GeneratedSerializationConstructorAccessor469(9000c730)>
<GC(760): unloaded and freed 1 class>
<GC(760): GC cycle started Mon Mar 9 11:36:02 2009
><GC(760): freed 9244952 bytes, 0% free (10540936/1073740288), in 337 ms>
<GC(760): mark: 256 ms, sweep: 13 ms, compact: 68 ms>
<GC(760): refs: soft 0 (age >= 32), weak 0, final 171, phantom 0>
<GC(760): moved 47010 objects, 20641336 bytes, IC reason=5>
<AF437: managing allocation failure, action=3 (10540936/1073740288)>
<AF437: managing allocation failure, action=4 (10540936/1073740288)>
<AF437: clearing all remaining soft refs>
<GC(761): GC cycle started Mon Mar 9 11:36:03 2009
><GC(761): freed 100328 bytes, 0% free (10641264/1073740288), in 329 ms>
<GC(761): mark: 230 ms, sweep: 14 ms, compact: 85 ms>
<GC(761): refs: soft 24 (age >= 32), weak 0, final 1, phantom 0>
<GC(761): moved 124854 objects, 44831488 bytes, IC reason=5>
<AF437: completed in 669 ms>

<AF438: Allocation Failure. need 65552 bytes, 6745 ms since last AF>
<AF438: managing allocation failure, action=2 (695880/1073740288)>
<GC(762): freeing class sun.reflect.GeneratedSerializationConstructorAccessor470(9002b9f0)>
<GC(762): unloaded and freed 1 class>
<GC(762): GC cycle started Mon Mar 9 11:36:10 2009
><GC(762): freed 11778704 bytes, 1% free (12474584/1073740288), in 367 ms>
<GC(762): mark: 262 ms, sweep: 13 ms, compact: 92 ms>
<GC(762): refs: soft 0 (age >= 32), weak 0, final 273, phantom 0>
<GC(762): moved 134987 objects, 57955208 bytes, IC reason=5>
<AF438: managing allocation failure, action=3 (12474584/1073740288)>
<AF438: managing allocation failure, action=4 (12474584/1073740288)>
<AF438: clearing all remaining soft refs>
<GC(763): GC cycle started Mon Mar 9 11:36:10 2009
><GC(763): freed 105288 bytes, 1% free (12579872/1073740288), in 319 ms>
<GC(763): mark: 232 ms, sweep: 14 ms, compact: 73 ms>
<GC(763): refs: soft 24 (age >= 32), weak 0, final 1, phantom 0>
<GC(763): moved 56499 objects, 26667024 bytes, IC reason=5>
<AF438: completed in 689 ms>
<GC(764): freeing class sun.reflect.GeneratedSerializationConstructorAccessor471(9000c730)>
<GC(764): unloaded and freed 1 class>

<GC(764): GC cycle started Mon Mar 9 11:36:17 2009
><GC(764): freed 5057904 bytes, 1% free (12374336/1073740288), in 1452 ms>
<GC(764): mark: 254 ms, sweep: 15 ms, compact: 1183 ms>
<GC(764): refs: soft 0 (age >= 32), weak 0, final 159, phantom 0>
<GC(764): moved 2739808 objects, 932954672 bytes, reason=4, used 6952 more bytes>

<AF439: Allocation Failure. need 528 bytes, 12003 ms since last AF>
<AF439: managing allocation failure, action=2 (0/1073740288)>
<GC(765): GC cycle started Mon Mar 9 11:36:22 2009
><GC(765): freed 10961288 bytes, 1% free (10961288/1073740288), in 203 ms>
<GC(765): mark: 190 ms, sweep: 13 ms, compact: 0 ms>
<GC(765): refs: soft 0 (age >= 32), weak 0, final 217, phantom 0>
<AF439: managing allocation failure, action=3 (10961288/1073740288)>
<AF439: managing allocation failure, action=4 (10961288/1073740288)>
<AF439: clearing all remaining soft refs>
<GC(766): GC cycle started Mon Mar 9 11:36:23 2009
><GC(766): freed 118936 bytes, 1% free (11080224/1073740288), in 291 ms>
<GC(766): mark: 228 ms, sweep: 14 ms, compact: 49 ms>
<GC(766): refs: soft 27 (age >= 32), weak 0, final 1, phantom 0>
<GC(766): moved 21238 objects, 8014744 bytes, IC reason=5>
<AF439: completed in 506 ms>

<AF440: Allocation Failure. need 65552 bytes, 1227 ms since last AF>
<AF440: managing allocation failure, action=2 (3040440/1073740288)>
<GC(767): freeing class sun.reflect.GeneratedSerializationConstructorAccessor472(9002b9f0)>
<GC(767): unloaded and freed 1 class>
<GC(767): GC cycle started Mon Mar 9 11:36:24 2009
><GC(767): freed 7486104 bytes, 0% free (10526544/1073740288), in 298 ms>
<GC(767): mark: 244 ms, sweep: 13 ms, compact: 41 ms>
<GC(767): refs: soft 0 (age >= 32), weak 0, final 104, phantom 0>
<GC(767): moved 1610 objects, 1387256 bytes, IC reason=5>
<AF440: managing allocation failure, action=3 (10526544/1073740288)>
<AF440: managing allocation failure, action=4 (10526544/1073740288)>
<AF440: clearing all remaining soft refs>
<GC(768): GC cycle started Mon Mar 9 11:36:24 2009
><GC(768): freed 95128 bytes, 0% free (10621672/1073740288), in 291 ms>
<GC(768): mark: 228 ms, sweep: 13 ms, compact: 50 ms>
<GC(768): refs: soft 22 (age >= 32), weak 0, final 1, phantom 0>
<GC(768): moved 11896 objects, 6294960 bytes, IC reason=5>
<AF440: completed in 592 ms>

Did set their ulimit (setenv.sh) to be :
#---------------------------------------------

1. Skipping JDK Check for current PeopleTools release.
2. At this time PeopleTools only requires a JDK. This may change.

#findJDK
#---------------------------------------------

1. Grab some file descriptors.

if ; then
if ; then
maxfiles=`ulimit -n`
else
maxfiles=`ulimit -H -n`
fi
if ; then
maxfiles=1024
fi
if ; then
maxfiles=1024
fi
ulimit -n 8192
fi

oh, one more thing their setting is, right now the jave heap is set to 1gb(min and max), used to be 2gb(min and max) and still weblogic crashed :

JAVA_OPTIONS_AIX="-Xms1024m -Xmx1024m"

env setup:
big-ip load balancer , 2 proxy servers, 5 web servers, 2 content servers and 5 application server domains

initial error :

PIA_weblogic.log:####<Feb 27, 2009 10:50:24 AM CST> <Emergency> <WebLogicServer> <ma1ua275> <PIA> <ListenThread.Default> <<WLS Kernel>> <BEA-000209> <ListenThread.run() failed: java.lang.OutOfMemoryError
PIA_weblogic.log:java.lang.OutOfMemoryError

min and max heap size is 1024mb and this server has 16Gb of RAM.

Any thoughts...>
 
Marcos Maia
Ranch Hand
Posts: 977
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

Under AIX you're probably using IBM jvm, so I'd suggest you to try to get help in websphere or IBM forums. About this kind of problem it's usually a memory leak in the app running under weblogic wich cuses it. You'll need a profile tool to monitor this IBM jvm you're using to find out where and when is the memory leak happening.

[]s
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic