Help!!!problem About Loading Test wso2 esb !

loris_jand's picture
Recently, We're making a loading test for wso2 esb, and the testing env: OS: SUSE Linux Enterprise Server 11 (x86_64) CPU: total 2 (cores per cpu 2, threads per core 1) Memory: 4k page, physical 3884960k(1024844k free), swap 2047k(2047k free) Java HotSpot(TM) Server VM (1.5.0_22-b03) for linux-x86 JAVA_OPTS: -Xms2048m -Xmx2048m -Xmn768m -XX:+UseConcMarkSweepGC we test two proxy services that proxy two actual soap style web services. we use Concurrent vm users about 400 , every time about 2 hours ,the jvm crashed,and the error log follow : any idea is helpful for me!thanks! # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0xf75f3d0f, pid=4892, tid=1769307024 # # Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode) # Problematic frame: # V [libjvm.so+0x1bcd0f] # --------------- T H R E A D --------------- Current thread (0x080e9648): ConcurrentMarkSweepThread [id=4926] siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x000000cc Registers: EAX=0x00000000, EBX=0xf7a00bd0, ECX=0xa2a1cf90, EDX=0xe8e054d0 ESP=0x69756d60, EBP=0x69756de8, ESI=0x69756da0, EDI=0x69756e70 EIP=0xf75f3d0f, CR2=0x000000cc, EFLAGS=0x00010216 Top of Stack: (sp=0x69756d60) 0x69756d60: e8e054d0 a2a1cf90 69756da0 69756e70 0x69756d70: 6cc70000 080875c0 9cc70000 0000000d 0x69756d80: 08060200 00000000 00000000 a6e4d930 0x69756d90: 9cc70000 15000000 0000133e f7f8b8dc 0x69756da0: f79e86a8 0815f830 080c90e8 9cc70000 0x69756db0: 15000000 080c911c 080c91dc 080c923c 0x69756dc0: a6e4db40 00000000 69756e70 69756e08 0x69756dd0: f7a00bd0 00000000 f7ea81d6 f7a00bd0 Instructions: (pc=0xf75f3d0f) 0xf75f3cff: 90 8b 41 04 83 ec 04 8d 50 08 8b 40 08 56 51 52 0xf75f3d0f: ff 90 cc 00 00 00 8b 83 10 07 00 00 83 c4 10 8b Stack: [0x696d7000,0x69758000), sp=0x69756d60, free space=511k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1bcd0f] V [libjvm.so+0x13a0bb] V [libjvm.so+0x1b634e] V [libjvm.so+0x1b617f] V [libjvm.so+0x1b4057] V [libjvm.so+0x1c062a] V [libjvm.so+0x43c008] C [libpthread.so.0+0x61b5] --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x6424f830 JavaThread "http-9443-Processor50" daemon [_thread_in_native, id=5105] 0x675c6318 JavaThread "http-9443-Processor49" daemon [_thread_blocked, id=5104] 0x675c5698 JavaThread "http-9443-Processor48" daemon [_thread_blocked, id=5103] 0x6675f240 JavaThread "http-9443-Processor47" daemon [_thread_blocked, id=5102] 0x645ce770 JavaThread "http-9443-Processor46" daemon [_thread_blocked, id=5101] 0x64266f70 JavaThread "http-9443-Processor45" daemon [_thread_blocked, id=5100] 0x63f1f4e0 JavaThread "http-9443-Processor44" daemon [_thread_blocked, id=5099] 0x6763cdb8 JavaThread "http-9443-Processor43" daemon [_thread_blocked, id=5098] 0x670f2538 JavaThread "http-9443-Processor42" daemon [_thread_blocked, id=5097] 0x673ebb08 JavaThread "http-9443-Processor41" daemon [_thread_blocked, id=5096] 0x668f8330 JavaThread "http-9443-Processor40" daemon [_thread_blocked, id=5095] 0x66a0f708 JavaThread "http-9443-Processor39" daemon [_thread_blocked, id=5094] 0x5d2d9540 JavaThread "http-9443-Processor38" daemon [_thread_blocked, id=5093] 0x6458f538 JavaThread "http-9443-Processor37" daemon [_thread_blocked, id=5092] 0x629f3790 JavaThread "http-9443-Processor36" daemon [_thread_in_native, id=5091] 0x5d9faaf0 JavaThread "http-9443-Processor35" daemon [_thread_in_native, id=5090] 0x67656128 JavaThread "http-9443-Processor34" daemon [_thread_in_native, id=5089] 0x645d7b98 JavaThread "http-9443-Processor33" daemon Other Threads: 0x081619f0 VMThread [id=4927] 0x67a3fca8 WatcherThread [id=4938] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap par new generation total 785728K, used 11267K [0x6cc70000, 0x9cc70000, 0x9cc70000) eden space 785024K, 1% used [0x6cc70000, 0x6d770c68, 0x9cb10000) from space 704K, 0% used [0x9cb10000, 0x9cb10000, 0x9cbc0000) to space 704K, 0% used [0x9cbc0000, 0x9cbc0000, 0x9cc70000) concurrent mark-sweep generation total 1310720K, used 712146K [0x9cc70000, 0xecc70000, 0xecc70000) concurrent-mark-sweep perm gen total 51712K, used 51563K [0xecc70000, 0xefef0000, 0xf0c70000) Dynamic libraries: 08048000-08057000 r-xp 00000000 08:01 341343 /usr/java/jdk1.5.0_22/bin/java 08057000-08059000 rwxp 0000e000 08:01 341343 /usr/java/jdk1.5.0_22/bin/java 08059000-09f0c000 rwxp 08059000 00:00 0 [heap] Environment Variables: JAVA_HOME=/usr/java/jdk1.5.0_22 PATH=/usr/java/jdk1.5.0_22/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin LD_LIBRARY_PATH=/usr/java/jdk1.5.0_22/jre/lib/i386/server:/usr/java/jdk1.5.0_22/jre/lib/i386:/usr/java/jdk1.5.0_22/jre/../lib/i386 SHELL=/bin/bash HOSTTYPE=x86_64 OSTYPE=linux MACHTYPE=x86_64-suse-linux
indika's picture

Hi All of your threads are

Hi All of your threads are 'http-9443-Processor*' .. are you using the servlet transport? Thanks Indika
loris_jand's picture

Sorry,for late

We invoke service,using 8280 port we found if we uncomment "-XX:+UseConcMarkSweepGC",it's work well the crashed thread is "ConcurrentMarkSweepThread" we're just tuning the wso2 esb3.0 thanks!
loris_jand's picture

http-9443-Processor

we still found many "colse_wait" connnections in our server side the thread is just "http-9443-Processor" when i close the esb web console ,the "close_wait" connections will disapper! ??????? thanks Indika!!!!
indika's picture

The 'http-9443-Processor'

The 'http-9443-Processor' should be the admin console HTTP transport. That is why when you closed the esb web console ,the "close_wait" connections disappeared. I suspect that the web console do some AJAX web service calls periodically,for example dashboard, statistics graphs,etc.(if you view them) Thanks Indika
library project main code
Learn Cloud
Learn
Cloud

The WSO2 Application Server is a reliable application server that can host your enterprise web applications. The WSO2 Application Server as a Service is offered in StratosLive, the WSO2 Platform as a Service. This article explains how a simple web application can be developed and deployed from Carbon Studio to the WSO2 Application Server...

Latest Webinar
Different groups within an organization need to monitor different Key Performance Indicators (KPIs) - An operations team will be interested in the response times of business services and loads of each service,..
Thursday, February 9th 2012, 09.00 AM (PST)

Thursday, February 9th 2012, 10.00 AM (GMT)