JBoss performance decrease

[Sury Bu]

Hi.all
When I doing a JBoss performance testing job,I find that the performance will decrease with time grow up.
I want to figure out what make it like this. Some information about my test list here:

1.Testing environment:

JBoss version:JBoss5.1.0.GA
Java version:JDK1.5
OS:Windows server 2003
DBMS:Sql server 2005
Testing tool: Apache Jmeter(i have set 50 threads and Ramp-Up Period in 30 seconds)

2.At first i have done some JBoss performance tuning.

a)I’ve add some parameters about JVM in thefile %JBOSS_HOME%bin\run.conf.bat

rem # JVM memory allocation pool parameters- modify as appropriate.

set "JAVA_OPTS=-Xms1024M -Xmx1024M-XX:MaxPermSize=256M -XX:NewSize=400M -XX:MaxNewSize=400M-XX:SurvivorRatio=32"

rem #

set -Djava.util.logging.manager=java.util.logging.LogManager-Djava.util.logging.config.file=C:\mcs\config\logging.properties"

b)Here is the logging.properties:

##########################################################

# Default Logging Configuration File

#

# You can use a different file byspecifying a filename

# with the java.util.logging.config.filesystem property.

# For example java-Djava.util.logging.config.file=myfile

############################################################



############################################################

# Global properties

############################################################



# "handlers" specifies a commaseparated list of log Handler

# classes. These handlers will be installed during VM startup.

# Note that these classes must be on thesystem classpath.

# By default we only configure aConsoleHandler, which will only

# show messages at the INFO and abovelevels.

handlers= java.util.logging.ConsoleHandler



# To also add the FileHandler, use thefollowing line instead.

#handlers= java.util.logging.FileHandler,java.util.logging.ConsoleHandler



# Default global logging level.

# This specifies which kinds of events arelogged across

# all loggers. For any given facility this global level

# can be overriden by a facility specificlevel

# Note that the ConsoleHandler also has aseparate level

# setting to limit messages printed to theconsole.

.level= OFF



############################################################

# Handler specific properties.

# Describes specific configuration info forHandlers.

############################################################



# default file output is in user's homedirectory.

java.util.logging.FileHandler.pattern =%h/java%u.log

java.util.logging.FileHandler.limit = 50000

java.util.logging.FileHandler.count = 1

java.util.logging.FileHandler.formatter =java.util.logging.XMLFormatter



# Limit the message that are printed on theconsole to INFO and above.

java.util.logging.ConsoleHandler.level =OFF

java.util.logging.ConsoleHandler.formatter= java.util.logging.SimpleFormatter





############################################################

# Facility specific properties.

# Provides extra control for each logger.

############################################################



# For example, set the com.xyz.foo loggerto only log SEVERE

# messages:

com.xyz.foo.level = SEVERE

com.microsoft.sqlserver.jdbc.internals.SQLServerResultSet= OFF

com.microsoft.sqlserver.jdbc.ResultSet =OFF

com.microsoft.sqlserver.jdbc = OFF

c)Add more system thread in "System thread pool size"

<!-- A Thread pool service -->

<mbean code="org.jboss.util.threadpool.BasicThreadPool"

name="jboss.system:service=ThreadPool">

<attribute name="Name">JBoss SystemThreads</attribute>

<attribute name="ThreadGroupName">SystemThreads</attribute>

<!-- How long a thread will live without any tasks in MS -->

<attribute name="KeepAliveTime">60000</attribute>

<!-- The max number of threads in the pool-->

<attributename="MaximumPoolSize">100</attribute>

<!-- The max number of tasks before the queue is full -->

<attributename="MaximumQueueSize">1000</attribute>

d)Set database pool size

<min-pool-size>100</min-pool-size>

<max-pool-size>300</max-pool-size>

Add parameters above into the *-ds filesunder %JBOSS_HOME%\server\all\deploy

[Sury Bu]

with the time increase JBossAvgProcessingTime become more and more large，finally become stabilized.
And then Throughput become more and more small finally become stabilized too.
What make this happen?

[Sury Bu]

Here is the GC data charps record by Jconsole come with JDK1.5. One houre later, major GC become so busy...and the Throughput in Jmeter become decrease with the time. I have increase the heap size above to 1300M. If more rhan 1300M, there will throw error ...So maybe in my enviroment the heap size limited to about 1.3G.
Should i increase the survior space?
Or there's something else going unnormal in my enviroment?
It looks like one hour later, one minor GC can only collect little die objects in survivor space, there are many objects are survivoring.
Then in the next minor GC time, lived objects fills up in the survivor space, any remaing live objects will copy to tenured generarion.
So we can see, major GC become more and more often then that in the first hour...
But what make me adoubt as why one hour later, the objects become difficult to release...
Can somebody help me to find out the problem with me?

Thanks!