JAVA HOT SPOT error in cognos 8.2

[uyravikumar]

We are getting JAVA HOT SPOT error and cognos is getting crashed very frequently.

Any hlep is appriciated.

AVA HOT SPOT Error Description
*************************
Environment details
***************
OS : Solaris spar 10
Application server : weblogic 8.6 sp1
Web server : sun one web server 6.1 sp4
Authentication provider : Java customer authentication provider
Installation type : Distributed (having 1 active content store and 3 standby content stores)
MEM_ARGS used : jvm_args: -Xms1024m -Xmx1024m -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=80 -Xnoclassgc -XX:+DisableExplicitGC -XX:MaxPermSize=512m
Cognos is getting crashed by giving the JAVA HOT SPOT error.

hs_err_pid12237.log contents are below
******************************************************************************
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x8d6d5f7c, pid=12237, tid=961
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_11-b06 mixed mode)
# Problematic frame:
# C  [libCCLCore.so+0xd5f7c]  __1cKI18NStringFempty6kM_b_+0x0
#

---------------  T H R E A D  ---------------

Current thread (0x01e43f90):  JavaThread "Thread-1115" [_thread_in_native, id=961]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x000001d4

Registers:
O0=0x000001d0 O1=0x00000034 O2=0x01f73218 O3=0x00000000
O4=0x85fe5150 O5=0x0130f248 O6=0x85fe4198 O7=0x8d6d967c
G1=0x00036000 G2=0x00000034 G3=0x01f73218 G4=0x00000003
G5=0x00000002 G6=0x00000000 G7=0x85fe9d98 Y=0x00000000
PC=0x8d6d5f7c nPC=0x8d6d5f80


Top of Stack: (sp=0x85fe4198)
0x85fe4198:   a610d848 85fe41a8 00000001 fe592000
0x85fe41a8:   01e43f90 00000000 8b3fe978 85fe5134
0x85fe41b8:   85fe51e8 000001d0 00000000 00000000
0x85fe41c8:   85fe51e8 000001d0 85fe4200 8b2a911c
0x85fe41d8:   00dd6f90 00dd6f90 01e43f90 01078af8
0x85fe41e8:   00dd6f88 00dd6f90 00dd737c 01078b58
0x85fe41f8:   01e43f90 01e43f90 00710ae4 00014400
0x85fe4208:   8b3f0710 00000000 fe5d4be4 01e44028

Instructions: (pc=0x8d6d5f7c)
0x8d6d5f6c:   00 00 00 00 c0 02 20 04 81 c3 e0 08 90 10 20 00
0x8d6d5f7c:   da 02 20 04 96 20 00 0d 95 32 f0 3f 80 9a a0 01

Stack: [0x85c02000,0x85fe9d98),  sp=0x85fe4198,  free space=3976k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libCCLCore.so+0xd5f7c]  __1cKI18NStringFempty6kM_b_+0x0
C  [libCAM_AAA_Engine.so+0x29124]
C  [libCAM_AAA_Engine.so+0x49f90]
C  [libCAM_AAA_Engine.so+0x78fb4]
C  [libCCLCore.so+0x4a8dc]  __1cJCCLThreadEmain6Fp0_pv_+0x64
C  [libCCLCore.so+0xeae80]


---------------  P R O C E S S  ---------------


Other Threads:
  0x000eda70 VMThread [id=4]
  0x000f70e8 WatcherThread [id=12]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation   total 174720K, used 31820K [0x99400000, 0xa60c0000, 0xa60c0000)
  eden space 139776K,  15% used [0x99400000, 0x9a8a6138, 0xa1c80000)
  from space 34944K,  30% used [0xa3ea0000, 0xa490d1d8, 0xa60c0000)
  to   space 34944K,   0% used [0xa1c80000, 0xa1c80000, 0xa3ea0000)
tenured generation   total 838912K, used 83599K [0xa60c0000, 0xd9400000, 0xd9400000)
   the space 838912K,   9% used [0xa60c0000, 0xab263f70, 0xab264000, 0xd9400000)
compacting perm gen  total 49152K, used 49108K [0xd9400000, 0xdc400000, 0xf9400000)
   the space 49152K,  99% used [0xd9400000, 0xdc3f5158, 0xdc3f5200, 0xdc400000)

Dynamic libraries:
0x00010000    /vol01/local/bea/wls81/jdk142_11/bin/java
0xff350000    /usr/lib/libthread.so.1
0xff340000    /usr/lib/libdl.so.1
0xff200000    /usr/lib/libc.so.1
0xff390000    /usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1
0xfe000000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/server/libjvm.so
0xff2d0000    /usr/lib/libCrun.so.1
0xff1e0000    /usr/lib/libsocket.so.1
0xff100000    /usr/lib/libnsl.so.1
0xff0b0000    /usr/lib/libm.so.1
0xff1b0000    /usr/lib/libsched.so.1
0xff090000    /usr/lib/libmp.so.2
0xff060000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/native_threads/libhpi.so
0xff040000    /usr/lib/nss_files.so.1
0xfe7d0000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/libverify.so
0xfe790000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/libjava.so
0xfe770000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/libzip.so
0xf9610000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/libnet.so
0xfc3a0000    /vol01/local/bea/wls81/weblogic/server/lib/solaris/libweblogicunix1.so
0xf9510000    /vol01/local/bea/wls81/weblogic/server/lib/solaris/libstackdump.so
0xf9460000    /vol01/local/bea/wls81/weblogic/server/lib/solaris/libmuxer.so
0xf9440000    /usr/ucblib/libucb.so.1
0x993b0000    /usr/lib/libresolv.so.2
0x992c0000    /usr/lib/libelf.so.1
0x99390000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/libnio.so
0x992a0000    /usr/lib/librt.so.1
0x99190000    /usr/lib/libaio.so.1
0x98f60000    /usr/lib/libsendfile.so.1
0x98f40000    /vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/libioser12.so
0x93510000    /vol01/local/cognos/c8/bin/libLogIPFConfigure.so
0x8dd80000    /usr/lib/libX11.so.4
0x93460000    /usr/lib/libgen.so.1
0x93430000    /usr/lib/libpthread.so.1
0x8db80000    /usr/lib/libCstd.so.1
0x93330000    /usr/lib/libiostream.so.1
0x93530000    /usr/lib/libintl.so.1
0x93250000    /usr/openwin/lib/libXext.so.0
0x93210000    /usr/lib/cpu/sparcv8plus/libCstd_isa.so.1
0x93310000    /vol01/local/cognos/c8/bin/libJCAM_Crypto_JNI.so
0x8d780000    /usr/local/cognos/c8/bin/libxerces-c.so.25
0x8d600000    /usr/local/cognos/c8/bin/libCCLCore.so
0x8d500000    /usr/local/cognos/c8/bin/libicucoguc.so.28
0x8cc00000    /usr/local/cognos/c8/bin/libicucogdata.so.28
0x93140000    /usr/local/cognos/c8/bin/libz.so
0x8ca80000    /usr/local/cognos/c8/bin/libicucogi18n.so.28
0x93060000    /usr/lib/nss_dns.so.1
0x8b480000    /vol01/local/cognos/c8/bin/libCMCacheLib.so
0x93020000    /usr/local/cognos/c8/bin/libCCLVirtualMemMgmt.so
0x8b280000    /usr/local/cognos/c8/bin/libCAM_AAA_Engine.so
0x8b100000    /usr/local/cognos/c8/bin/libCAM_Crypto_Interface.so
0x92f30000    /usr/local/cognos/c8/bin/libCCLCFGAPI.so
0x92e10000    /usr/local/cognos/c8/bin/libCCLIDOM.so
0x92d40000    /usr/local/cognos/c8/bin/libcogipf2.so
0x8b000000    /usr/local/cognos/c8/bin/coglog4ccl.so
0x92d10000    /usr/local/cognos/c8/bin/libIBJStreamsDLL.so
0x92c50000    /usr/local/cognos/c8/bin/libCCLHttptools.so
0x92f10000    /vol01/local/cognos/c8/bin/libFMMD_UpgradeCM.so
0x92c20000    /usr/local/cognos/c8/bin/libFMMD_Resources.so
0x92b60000    /usr/local/cognos/c8/bin/libFMMD_UpgradeCommon.so
0x8aa80000    /usr/local/cognos/c8/bin/libxalan-c.so.18
0x8a980000    /usr/local/cognos/c8/bin/libBMTUtil.so
0x92b30000    /usr/local/cognos/c8/bin/libFMMD_XSLTransformer.so
0x92b10000    /usr/local/cognos/c8/bin/libxalanMsg.so.18
0x92a60000    /usr/local/cognos/c8/bin/libIBJHelpers.so
0x92a40000    /usr/local/cognos/c8/bin/libBmtResources.so
0x8a880000    /usr/local/cognos/c8/bin/libQEUtility.so
0x91fa0000    /usr/local/cognos/c8/bin/libUTUtilities.so
0x8a780000    /usr/local/cognos/c8/bin/libzbstore.so
0x92560000    /vol01/local/cognos/c8/bin/libCAM_AAA_JNI.so
0x92540000    /vol01/local/cognos/c8/bin/libCM_API.so
0x923e0000    /vol01/local/cognos/c8/./bin/libLogTypedFactory.so
0x92160000    /vol01/local/cognos/c8/./bin/libLogAuditRenderer.so
0x91de0000    /usr/local/cognos/c8/bin/libLogIndicationRenderer.so
0x91dc0000    /vol01/local/cognos/c8/./bin/libLogAuditAccessRenderer.so
0x91ce0000    /vol01/local/cognos/c8/./bin/libLogPerfRenderer.so
0x91cc0000    /vol01/local/cognos/c8/./bin/libLogTraceRenderer.so
0x91ca0000    /vol01/local/cognos/c8/./bin/libLogValueTraceRenderer.so
0x8a100000    /vol01/local/cognos/c8/./bin/libCAM_Crypto_POpenSSL.so
0x89f00000    /vol01/local/cognos/c8/./bin/libCAM_Crypto_TOpenSSL.so
0x89e00000    /vol01/local/cognos/c8/./bin/libCAM_AAA_Auth3rdParty.so

VM Arguments:
jvm_args: -Xms1024m -Xmx1024m -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=80 -Xnoclassgc -XX:+DisableExplicitGC -XX:MaxPermSize=512m -Dweblogic.security.SSL.trustedCAKeyStore=/lopt/bea81sp3/weblogic81/server/lib/cacerts -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dweblogic.management.discover=false -Dweblogic.Name=vcps1001 -Dweblogic.management.username= -Dweblogic.management.password= -Dweblogic.management.server=http://ustrsders008.dev.gweb.eds.com:7366 -Djava.security.policy=/usr/local/bea/wls81/weblogic/server/lib/weblogic.policy -Dgwh.common.dir=/sites/vcps1/site/common -Duser.country=US
java_command: weblogic.Server
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/local/bea/wls81/jdk142_11
CLASSPATH=/usr/local/bea/wls81/weblogic/server/lib/ojdbc14.jar:/usr/local/bea/wls81/jdk142_11/lib/tools.jar:/usr/local/bea/wls81/weblogic/server/lib/weblogic_sp.jar:/usr/local/bea/wls81/weblogic/server/lib/weblogic.jar::/usr/local/bea/wls81/weblogic/common/eval/pointbase/lib/pbserver44.jar:/usr/local/bea/wls81/weblogic/common/eval/pointbase/lib/pbclient44.jar:/usr/local/bea/wls81/jdk142_11/jre/lib/rt.jar:/usr/local/bea/wls81/weblogic/server/lib/webservices.jar:/sites/vcps1/site/common/wls81/lib:/usr/local/bea/wls81/weblogic/server/lib/weblogic_sp.jar:/usr/local/bea/wls81/weblogic/server/lib/weblogic.jar
PATH=/usr/local/bea/wls81/weblogic/server/bin:/usr/local/bea/wls81/jdk142_11/jre/bin:/usr/local/bea/wls81/jdk142_11/bin:/usr/local/bea/wls81/jdk142_11/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/ccs/bin:/usr/local/bin:/opt/openssh/bin:/opt/ipf/bin:/usr/local/oracle/product/latest/bin:/usr/openwin/bin:/opt/VRTS/bin:/opt/VRTSob/bin:/etc/vx/bin:/usr/bin:/bin:/usr/sbin:/usr/local/bin:/usr/sfw/bin:/usr/bin:/bin:/usr/sbin:/usr/local/bin
LD_LIBRARY_PATH=/vol01/local/bea/wls81/jdk142_11/jre/lib/sparc/server:/vol01/local/bea/wls81/jdk142_11/jre/lib/sparc:/vol01/local/bea/wls81/jdk142_11/jre/../lib/sparc::/usr/local/bea/wls81/weblogic/server/lib/solaris:/usr/local/bea/wls81/weblogic/server/lib/solaris/oci920_8:/usr/local/cognos/c8/bin:/usr/local/oracle/product/10.2.0/lib32:/usr/local/bea/wls81/weblogic/server/lib/solaris;
SHELL=/bin/ksh
DISPLAY=localhost:10.0


---------------  S Y S T E M  ---------------

OS:                       Solaris 8 2/04 s28s_hw4wos_05a SPARC
           Copyright 2004 Sun Microsystems, Inc.  All Rights Reserved.
                            Assembled 08 January 2004

uname:SunOS 5.8 Generic_117350-38 sun4u  (T1 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 1024, AS infinity
load average:0.01 0.01 0.02

CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

Memory: 8k page, physical 2097152k(669984k free)

vm_info: Java HotSpot(TM) Server VM (1.4.2_11-b06) for solaris-sparc, built on Feb 13 2006 10:37:03 by unknown with Workshop 5.2 compat=5



******************************************************************************

Thanks in advance.
Ravi.

[kolonell]

Hi Ravi

I would suggest reducing the Java Memory Arguments you use to start this instance (eg -Xmx768 , etc   would be a good place to start)
With only 2 GB (if I can read correctly) of RAM this is too high. I would even try to go for -Xmx512.

Another thing you might try is to add the -server option to the java options.

[uyravikumar]

Thx i will reduce the mem_args..


I have set JAVA_VM=-server in startmanagedweblogic.script.

Can you pls send -server option detail i dont have.

Thank you
Ravi.

[kolonell]

setting it in the script ought to be enough but I am not sure as I don't see the option reflected in the jvm_args listed in the java core.

I 'll check my script as soon as i get to the office ;-).

[kolonell]

when executing the script startManagedWebLogic.cmd <servername> verify the java parameters that get echoed on the command line.
mine looks like this : (you can see the -server option is the first one in the list)

E:\bea\JDK150~1\bin\java -server -Xms256m -Xmx512m  -XX:MaxPermSize=128m -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dweblogic.security.SSL.trustedCAKeyStore="E:\bea\weblogic92\server\lib\cacerts"  -Xverify:none  -da -Dplatform.home=E:\bea\WEBLOG~1 -Dwls.home=E:\bea\WEBLOG~1\server -Dwli.home=E:\bea\WEBLOG~1\integration  -Dweblogic.management.discover=false -Dweblogic.management.server=http://localhost:7001  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=E:\bea\patch_weblogic920\profiles\default\sysext_manifest_classpath -Dweblogic.management.username= -Dweblogic.management.password=  -da -Dplatform.home=E:\bea\WEBLOG~1 -Dwls.home=E:\bea\WEBLOG~1\server -Dwli.home=E:\bea\WEBLOG~1\integration  -Dweblogic.management.discover=false -Dweblogic.management.server=http://localhost:7001  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=E:\bea\patch_weblogic920\profiles\default\sysext_manifest_classpath -Dweblogic.Name=EPServer -Djava.security.policy=E:\bea\WEBLOG~1\server\lib\weblogic.policy  weblogic.Server

[uyravikumar]

Hi,

I reduced the mem_args and it is given below. Even after that only one dispatcher got crashed.

SET MEM_ARGS=XX:NewSize=300m -XX:MaxNewSize=300m -Xms768m -Xmx1024m -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=80 -Xnoclassgc -XX:+DisableExplicitGC -XX:MaxPermSize=128m"

I did not see the -server option coming in my startmanagedweblogic.out file when i am starting the weblogic.

We have similar crash file in content and size after reducing the MEM_ARGS.

Please let me know any other options..i am also trying other options.

Thank you
Ravi.

[kolonell]

Just set the MEM_ARGS variable to the following :  -Xms512m -Xmx768m  -XX:MaxPermSize=128m

I typically let the java vm handle the other variables as they can detriment performance if you don't have a good knowledge
of memory management in java.

[uyravikumar]

Hi all,

We raised a ticket on cognos and cognos is providing a Hot fix for Java hotspot error.

the current version of cognos we are using is "8.2.43.128".

Regards
Ravi.

[aero_terrio]

Hi All,

Was there a resolution to this issue? Did the cognos hot fix do the trick? We are starting to see the exact same behaviour with our environment currently so I would be interested in hearing the result.

Thanks