Help!!!problem About Loading Test wso2 esb !

loris_jand's picture
Recently, We're making a loading test for wso2 esb, and the testing env: OS: SUSE Linux Enterprise Server 11 (x86_64) CPU: total 2 (cores per cpu 2, threads per core 1) Memory: 4k page, physical 3884960k(1024844k free), swap 2047k(2047k free) Java HotSpot(TM) Server VM (1.5.0_22-b03) for linux-x86 JAVA_OPTS: -Xms2048m -Xmx2048m -Xmn768m -XX:+UseConcMarkSweepGC we test two proxy services that proxy two actual soap style web services. we use Concurrent vm users about 400 , every time about 2 hours ,the jvm crashed,and the error log follow : any idea is helpful for me!thanks! # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0xf75f3d0f, pid=4892, tid=1769307024 # # Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode) # Problematic frame: # V [libjvm.so+0x1bcd0f] # --------------- T H R E A D --------------- Current thread (0x080e9648): ConcurrentMarkSweepThread [id=4926] siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x000000cc Registers: EAX=0x00000000, EBX=0xf7a00bd0, ECX=0xa2a1cf90, EDX=0xe8e054d0 ESP=0x69756d60, EBP=0x69756de8, ESI=0x69756da0, EDI=0x69756e70 EIP=0xf75f3d0f, CR2=0x000000cc, EFLAGS=0x00010216 Top of Stack: (sp=0x69756d60) 0x69756d60: e8e054d0 a2a1cf90 69756da0 69756e70 0x69756d70: 6cc70000 080875c0 9cc70000 0000000d 0x69756d80: 08060200 00000000 00000000 a6e4d930 0x69756d90: 9cc70000 15000000 0000133e f7f8b8dc 0x69756da0: f79e86a8 0815f830 080c90e8 9cc70000 0x69756db0: 15000000 080c911c 080c91dc 080c923c 0x69756dc0: a6e4db40 00000000 69756e70 69756e08 0x69756dd0: f7a00bd0 00000000 f7ea81d6 f7a00bd0 Instructions: (pc=0xf75f3d0f) 0xf75f3cff: 90 8b 41 04 83 ec 04 8d 50 08 8b 40 08 56 51 52 0xf75f3d0f: ff 90 cc 00 00 00 8b 83 10 07 00 00 83 c4 10 8b Stack: [0x696d7000,0x69758000), sp=0x69756d60, free space=511k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1bcd0f] V [libjvm.so+0x13a0bb] V [libjvm.so+0x1b634e] V [libjvm.so+0x1b617f] V [libjvm.so+0x1b4057] V [libjvm.so+0x1c062a] V [libjvm.so+0x43c008] C [libpthread.so.0+0x61b5] --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x6424f830 JavaThread "http-9443-Processor50" daemon [_thread_in_native, id=5105] 0x675c6318 JavaThread "http-9443-Processor49" daemon [_thread_blocked, id=5104] 0x675c5698 JavaThread "http-9443-Processor48" daemon [_thread_blocked, id=5103] 0x6675f240 JavaThread "http-9443-Processor47" daemon [_thread_blocked, id=5102] 0x645ce770 JavaThread "http-9443-Processor46" daemon [_thread_blocked, id=5101] 0x64266f70 JavaThread "http-9443-Processor45" daemon [_thread_blocked, id=5100] 0x63f1f4e0 JavaThread "http-9443-Processor44" daemon [_thread_blocked, id=5099] 0x6763cdb8 JavaThread "http-9443-Processor43" daemon [_thread_blocked, id=5098] 0x670f2538 JavaThread "http-9443-Processor42" daemon [_thread_blocked, id=5097] 0x673ebb08 JavaThread "http-9443-Processor41" daemon [_thread_blocked, id=5096] 0x668f8330 JavaThread "http-9443-Processor40" daemon [_thread_blocked, id=5095] 0x66a0f708 JavaThread "http-9443-Processor39" daemon [_thread_blocked, id=5094] 0x5d2d9540 JavaThread "http-9443-Processor38" daemon [_thread_blocked, id=5093] 0x6458f538 JavaThread "http-9443-Processor37" daemon [_thread_blocked, id=5092] 0x629f3790 JavaThread "http-9443-Processor36" daemon [_thread_in_native, id=5091] 0x5d9faaf0 JavaThread "http-9443-Processor35" daemon [_thread_in_native, id=5090] 0x67656128 JavaThread "http-9443-Processor34" daemon [_thread_in_native, id=5089] 0x645d7b98 JavaThread "http-9443-Processor33" daemon Other Threads: 0x081619f0 VMThread [id=4927] 0x67a3fca8 WatcherThread [id=4938] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap par new generation total 785728K, used 11267K [0x6cc70000, 0x9cc70000, 0x9cc70000) eden space 785024K, 1% used [0x6cc70000, 0x6d770c68, 0x9cb10000) from space 704K, 0% used [0x9cb10000, 0x9cb10000, 0x9cbc0000) to space 704K, 0% used [0x9cbc0000, 0x9cbc0000, 0x9cc70000) concurrent mark-sweep generation total 1310720K, used 712146K [0x9cc70000, 0xecc70000, 0xecc70000) concurrent-mark-sweep perm gen total 51712K, used 51563K [0xecc70000, 0xefef0000, 0xf0c70000) Dynamic libraries: 08048000-08057000 r-xp 00000000 08:01 341343 /usr/java/jdk1.5.0_22/bin/java 08057000-08059000 rwxp 0000e000 08:01 341343 /usr/java/jdk1.5.0_22/bin/java 08059000-09f0c000 rwxp 08059000 00:00 0 [heap] Environment Variables: JAVA_HOME=/usr/java/jdk1.5.0_22 PATH=/usr/java/jdk1.5.0_22/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin LD_LIBRARY_PATH=/usr/java/jdk1.5.0_22/jre/lib/i386/server:/usr/java/jdk1.5.0_22/jre/lib/i386:/usr/java/jdk1.5.0_22/jre/../lib/i386 SHELL=/bin/bash HOSTTYPE=x86_64 OSTYPE=linux MACHTYPE=x86_64-suse-linux
indika's picture

Hi All of your threads are

Hi All of your threads are 'http-9443-Processor*' .. are you using the servlet transport? Thanks Indika
loris_jand's picture

Sorry,for late

We invoke service,using 8280 port we found if we uncomment "-XX:+UseConcMarkSweepGC",it's work well the crashed thread is "ConcurrentMarkSweepThread" we're just tuning the wso2 esb3.0 thanks!
loris_jand's picture

http-9443-Processor

we still found many "colse_wait" connnections in our server side the thread is just "http-9443-Processor" when i close the esb web console ,the "close_wait" connections will disapper! ??????? thanks Indika!!!!
indika's picture

The 'http-9443-Processor'

The 'http-9443-Processor' should be the admin console HTTP transport. That is why when you closed the esb web console ,the "close_wait" connections disappeared. I suspect that the web console do some AJAX web service calls periodically,for example dashboard, statistics graphs,etc.(if you view them) Thanks Indika