*
The moose likes Threads and Synchronization and the fly likes Multithreading and Memory Management Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Multithreading and Memory Management" Watch "Multithreading and Memory Management" New topic
Author

Multithreading and Memory Management

saket rustagi
Greenhorn

Joined: Mar 05, 2005
Posts: 1
Hi Everybody,

I am facing some problems while creating large number of threads (~ 10 lakhs).

JAVA version I am using is:
java version "1.2.2-12"
Classic VM (build J2SDK.v.1.2.2:08/14/2001-17:00, native threads, jit_122)


Command I used to run my program on COMPAQ (OS - UNIX) is:
java -Xmx1024m -Xms1024m -classpath .:*.jar dom.RadiusSimulation > test

In my main program, I continuously create threads til the number reaches 10 lakhs.
I create 10 threads(approx) every second.
Processing/Life time for every thread is approximately 3 seconds.

After creating every 1000 threads, I run
System.runFinalization(); and System.gc();
to free memory explicitly.

See the excerpts after the creation of first 1000 threads

Start Executing Finalization --> System.runFinalization()
Stop Executing Finalization --> System.runFinalization()
Start Executing Garbage Collector --> System.gc()
free memory before running Garbage Collector: 860124192
Stop Executing Garbage Collector --> System.gc()
free memory after running Garbage Collector: 941733792

See the excerpts after the creation of first 10000 threads

Start Executing Finalization --> System.runFinalization()
Stop Executing Finalization --> System.runFinalization()
Start Executing Garbage Collector --> System.gc()
free memory before running Garbage Collector: 632287632
Stop Executing Garbage Collector --> System.gc()
free memory after running Garbage Collector: 832233552

See the excerpts after the creation of first 60000 threads

Start Executing Finalization --> System.runFinalization()
Stop Executing Finalization --> System.runFinalization()
Start Executing Garbage Collector --> System.gc()
free memory before running Garbage Collector: 188833120
Stop Executing Garbage Collector --> System.gc()
free memory after running Garbage Collector: 285217328


These figures of free memory before and after running the System.gc() continuously decreases.
This seems to be a problem.

I don't know why this is happening?

Looks as if when System.gc() runs, it is not
able to completely free the memory occupied by a thread. So each thread starts hogging some part of
memory and at the end memory becomes too less to execute or continue the program.

Do anybody have some clue how to solve this problem and why it is happening?


Just after executing around 70,000 threads I start geting problems related to memory. For ex,
java.lang.OutOfMemoryError
at com.theorem.radius3.RADIUSClient.setDebug(RADIUSClient.java, Compiled Code)
at com.theorem.radius3.RADIUSClient.setDebug(RADIUSClient.java, Compiled Code)
at dom.Reader.run(Reader.java, Compiled Code)
at java.lang.Thread.run(Thread.java:479)


As I reach the value of 80,000 threads, frequency of above error increases,
and increases even more as we progress and at the end when we reach the figure around 75380,
JVM crashes with the following message:

SIGSEGV 11* segmentation violation
si_signo [11]: SIGSEGV 11* segmentation violation
si_errno [0]: Successful
si_code [1]: SEGV_MAPERR [addr: 0x0]

sc_pc: 0x300000244d8, r26: 0x30000024760

stackpointer=2000324cf28

Full thread dump Classic VM (J2SDK.v.1.2.2:08/14/2001-17:00, native threads):
"thread75380" (TID:0x8c5b560, sys_thread_t:0x40760cf8, state:MW, native ID:0x3c5f440) prio=5
at com.theorem.radius3.AttributeList.loadRadiusAttributes(AttributeList.java:529)
at com.theorem.radius3.RADIUSClient.recvPacket(RADIUSClient.java:1621)
at com.theorem.radius3.RADIUSClient.authenticate(RADIUSClient.java:791)
at com.theorem.radius3.RADIUSClient.authenticate(RADIUSClient.java:667)
at dom.Reader.run(Reader.java:89)
at java.lang.Thread.run(Thread.java:479)
"thread75379" (TID:0x8c83010, sys_thread_t:0x4081dcf8, state:MW, native ID:0x3757440) prio=5
at dom.TagSearch.<init>(TagSearch.java:9)
at dom.Reader.fill_RadiusMessage(Reader.java:595)
at dom.Reader.run(Reader.java:89)
at java.lang.Thread.run(Thread.java:479)
"thread75378" (TID:0x8c7f1b0, sys_thread_t:0x408160f8, state:R, native ID:0x324f440) prio=5
at com.theorem.radius3.radutil.RadRand.<init>(RadRand.java:24)
at com.theorem.radius3.RADIUSClient.coreInit(RADIUSClient.java:428)
at com.theorem.radius3.RADIUSClient.<init>(RADIUSClient.java:386)
at dom.Reader.run(Reader.java:89)
at java.lang.Thread.run(Thread.java:479)
"thread75377" (TID:0x8c58080, sys_thread_t:0x4081d0f8, state:MW, native ID:0x283f440) prio=5
at dom.TagSearch.<init>(TagSearch.java:9)
at dom.Reader.compare_RadiusResponseMessage(Reader.java:855)
at dom.Reader.run(Reader.java:89)
at java.lang.Thread.run(Thread.java:479)
"thread75376" (TID:0x8c783e0, sys_thread_t:0x408260f8, state:MW, native ID:0x2337440) prio=5
at com.theorem.radius3.RADIUSClient.duplicateExists(RADIUSClient.java:2332)
at com.theorem.radius3.RADIUSClient.recvPacket(RADIUSClient.java:1621)
at com.theorem.radius3.RADIUSClient.accounting(RADIUSClient.java:967)
at dom.Reader.run(Reader.java:89)
at java.lang.Thread.run(Thread.java:479)
"thread75375" (TID:0x8c3b940, sys_thread_t:0x408200f8, state:MW, native ID:0x2d47440) prio=5
at dom.TagSearch.<init>(TagSearch.java:9)
at dom.Reader.compare_RadiusResponseMessage(Reader.java:855)
at dom.Reader.run(Reader.java:89)
at java.lang.Thread.run(Thread.java:479)
"Finalizer" (TID:0x18880, sys_thread_t:0x403940f8, state:MW, native ID:0x1e2f440) prio=8
at com.theorem.radius3.RADIUSClient.finalize(RADIUSClient.java, Compiled Code)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:90)
at java.lang.ref.Finalizer.access$1(Finalizer.java:82)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:175)
"Reference Handler" (TID:0x180e0, sys_thread_t:0x402df8f8, state:CW, native ID:0x1927440) prio=10
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:424)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:114)
"Signal dispatcher" (TID:0x18120, sys_thread_t:0x402df4f8, state:CW, native ID:0xf17440) prio=5
"main" (TID:0x18240, sys_thread_t:0x402520f8, state:MW, native ID:0xffffffffc01b6000) prio=5
at java.lang.String.<init>(String.java:232)
at java.lang.Long.toString(Long.java:92)
at java.lang.String.valueOf(String.java:2038)
at java.lang.StringBuffer.append(StringBuffer.java:527)
at dom.RadiusSimulation.main(RadiusSimulation.java, Compiled Code)
Monitor Cache Dump:
dom.Reader@8C82DB0/3FFE3A40: owner "thread75379" (0x4081dcf8) 1 entry
java.lang.ref.Reference$Lock@180F0/CCE8D00: owner "thread75378" (0x408160f8) 1 entry
Waiting to be notified:
"Reference Handler" (0x402df8f8)
Registered Monitor Dump:
utf8 hash table: <unowned>


Here is some relevant portion from my code.


while(true)
{
// for loop will create 10 threads per second
for (int k=0; k < 1000 ;thread_no++, k=k+100)
{
try{
// Reader Class implements Runnable
// every time we create an object of Reader we create a new thread.
new Reader(document, S_test_no, radiusServer, radiusSecret, auport, accport, debug, ("thread" + thread_no), true);
} catch ( Exception ne)
{
System.out.println("NullPointerException Caught : " + ne + "\n Unable to create thread");
}
try {
// Main thread will sleep i.e. will not create thread for this much
Thread.sleep(100);
} catch (InterruptedException e)
{
System.out.println("InterruptedException Caught : " + e);
}
if(thread_no >= 1000000)
{
System.out.println("wait for other threads to return");
break;// stop the creation of new threads
}
if( thread_no % 1000 == 0 )
{
Runtime r = Runtime.getRuntime();
System.out.println(" Start Executing Finalization --> System.runFinalization() ");
System.runFinalization();
System.out.println(" Stop Executing Finalization --> System.runFinalization() ");
System.out.println(" Start Executing Garbage Collector --> System.gc() ");
System.out.println("free memory before running Garbage Collector: " + r.freeMemory());
System.gc();
System.out.println(" Stop Executing Garbage Collector --> System.gc() ");
System.out.println("free memory after running Garbage Collector: " + r.freeMemory());
}
}//for end

} //while end


Please give your comments as soon as possible.
Thanks Every body.

Saket

[EJFH: Everyone thinks their post is "URGENT." Saying so in the subject just prevents some of the words in the subject from appearing on the saloon front page, so fewer people will be likely to help you. ]
[ March 05, 2005: Message edited by: Ernest Friedman-Hill ]
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
There are good alternatives to setting up and tearing down so many threads, created to solve just such problems. If you're on JDK 5 look at Executor and Executors for some ideas. If you're on an earlier JDK, look for a nice open source thread pool like Apache Commons. If you start a thousand a second and they run three seconds you'll have about 3000 running concurrently which seems like a very big number to me. But with a pool you'll only have to start each thread once so you'll save nearly a million thread creates over the program.


A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
William Barnes
Ranch Hand

Joined: Mar 16, 2001
Posts: 984

1] Is there a real reason you need to create these many threads, or is this just for fun?

2] Exactly why is this specific post "urgent"?

3] What is a "lakhs"? Or how many is a "lakhs".


Please ignore post, I have no idea what I am talking about.
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
lakh == 100,000. While I'm here, I'll vote again for thread pooling, and for trying a smaller thread pool than 3,000 - maybe start with 50 - just to see what happens to performance.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Multithreading and Memory Management
 
Similar Threads
Thread dump - thread states
main app won't stop when all threads are done running
Thread Dump while using sockets
Monitoring threads in WebSphere 3.5 ??
UNIXProcess.forkandExec() hang?