Cognos hangs frequently. Need to re-start the service to fix this.

[nsaxena]

Hi All ,

I am facing a serious problem of cognos. I joined this project where cognos was installed already (by someone else) and he left.

We are using cognos 10.2.1 version with db2 as database. Now usually what happen is that after some runs ...cognos hangs...i.e none of the report works. Every report keeps on running state.

To fix this issue we re-start the cognos service . But this issue is re-occurring.  I have no clue about admin and there is no dba/cognos admin in my team. So i need to fix this issue. Kindly advice suggest what is causing this issue and a probable fix.

Please let me know if you need any screenshot of the cognos config file or similar stuff to know in moe detail about the issue.

Many thanks in advance!
Nisha

[MFGF]

Hi All ,

I am facing a serious problem of cognos. I joined this project where cognos was installed already (by someone else) and he left.

We are using cognos 10.2.1 version with db2 as database. Now usually what happen is that after some runs ...cognos hangs...i.e none of the report works. Every report keeps on running state.

To fix this issue we re-start the cognos service . But this issue is re-occurring.  I have no clue about admin and there is no dba/cognos admin in my team. So i need to fix this issue. Kindly advice suggest what is causing this issue and a probable fix.

Please let me know if you need any screenshot of the cognos config file or similar stuff to know in moe detail about the issue.

Many thanks in advance!
Nisha

Hi,

Are you seeing any errors in cogserver.log? That would be the first place to look. If I was a betting man I'd wager the java servlet container is running out of memory. What heap size is set in Cognos Configuration?

Also, it's a SERIOUSLY bad idea to have a live Cognos instance without a knowledgeable person administering it. People in the business depend on it to deliver the answers to key business questions, and make critical decisions based on what they see. You should push for yourself or someone else on the team to get trained up immediately on the Admin functions and best practices.

MF.

[nsaxena]

Hi MF,

How to check heap size.

Also i came to know below link. Is this helpful? I mean do i need to change settings in startup.bat file?
http://www.cisco.com/c/en/us/td/docs/net_mgmt/datacenter_mgmt/intel_auto/service_portal/v_10_0_r2/installation/guide/CiscoPrimeServiceCatalog-10-0-R2-InstallationGuide/cognos-troubleshooting.pdf

server machine ram on which cognos is installed is 8 gb.

[sdf]

In cognos connection goto Cognos Administration >> Configuration >> Dispatchers and Services >> Select your server>>Actions>>view services>>look for query services >> set properties >>settings

Check JVM set

In cognos configuration under cognos service. default value is set to 768.

[nsaxena]

Hi MF/sdf ,

Thanks for sharing details for checking. It is set to 1024 .I am attaching the screenshot for the same.

Please kindly advice me for future checks/suggestions in order to resolve this issue.

Many thanks in advance!

[nsaxena]

Hi MFGF,

As you suggested ,i checked the cognos server log. Only thing i could find catch is below lines in file (mentioned in last). Because of this 1 gb .dmp file is getting created after every re-start. We need to do 2-3 times cognos re-start every day :(

Also i am attaching the log file for your reference. It looks like JVM issue,but what and how to fix its above my head :(

IBM KB i could find this link :http://www-01.ibm.com/support/docview.wss?uid=swg21698578
which is showing similar issue(not exactly same). Kindly advice me how to proceed.

I am using cognos version as 10.2.1.


Few lines from cognos server logs :-

"
QueryService failed to start with the following command-line arguments: [C:\Program Files\IBM\cognos\c10_64\bin64\jre

\7.0\bin\java.exe, -Xms1024m, -Xmx1024m, -Xmnx512m, -Xmns512m, -server, -Xdisableexplicitgc, -Xscmx50m, -Xshareclasses:name=cognos10%u,nonfatal, -Xmso512K, -

Xcompressedrefs, -verbose:gc, -Xverbosegclog:C:/Program Files/IBM/cognos/c10_64/logs\XQE\dq_verbosegc_%Y%m%d.%H%M%S.%pid.log,1,1000, -Xgcpolicy:gencon, -

Dcubingservices_home=C:/Program Files/IBM/cognos/c10_64/v5dataserver, -DParentProcessPort=49340, -Djava.library.path=C:/Program Files/IBM/cognos/c10_64/bin64;, -

classpath, C:/Program Files/IBM/cognos/c10_64/webapps/p2pd/WEB-INF/classes;C:\Program Files\IBM\cognos\c10_64\v5dataserver\lib\*;C:\Program Files\IBM\cognos

\c10_64\v5dataserver\lib\ext\*;C:\Program Files\IBM\cognos\c10_64\webapps\p2pd\WEB-INF\lib\*;C:\Program Files\IBM\cognos\c10_64\tomcat\lib\*;, -DenableTraceServer, -

ea, -Dorg.dom4j.QName.singleton.strategy=org.dom4j.util.PerThreadSingleton, -Dorg.dom4j.DocumentFactory.singleton.strategy=org.dom4j.util.PerThreadSingleton,

com.cognos.xqe.cubingservices.DQServer] with console output: 08:56:59.359 0x416b0100   j9shr.1009   *   ** ASSERTION FAILED ** at CacheMap.cpp:1014: ((0 )) 

JVMDUMP039I Processing dump event "traceassert", detail "" at 2015/09/15 14:26:59 - please wait.  JVMDUMP032I JVM requested System dump using 'C:\Program Files\IBM

\cognos\c10_64\bin64\core.20150915.142659.5660.0001.dmp' in response to an event  JVMDUMP010I System dump written to C:\Program Files\IBM\cognos

\c10_64\bin64\core.20150915.142659.5660.0001.dmp  JVMDUMP032I JVM requested Java dump using 'C:\Program Files\IBM\cognos

\c10_64\bin64\javacore.20150915.142659.5660.0002.txt' in response to an event  JVMDUMP010I Java dump written to C:\Program Files\IBM\cognos

\c10_64\bin64\javacore.20150915.142659.5660.0002.txt  JVMDUMP032I JVM requested Snap dump using 'C:\Program Files\IBM\cognos

\c10_64\bin64\Snap.20150915.142659.5660.0003.trc' in response to an event  JVMDUMP010I Snap dump written to C:\Program Files\IBM\cognos\c10_64\bin64\Snap.20150915.142659.5660.0003.trc  JVMDUMP013I Processed dump event "traceassert", detail "".
"

[nsaxena]

Adding to above comment, from cognos configuration tool,while re-starting the cognos service,only error message which comes up is :-

'QueryService', 'StartService', 'Success'.
16:28:05, 'com.cognos.pogo.reportservice.ProcessManager', 'pogo', 'Failure'.
DPR-DPR-1035 Dispatcher detected an error.
Accept timed out

Is it related?

[sunosoft]

What is the value for JVM setting in your configuration ? How much is RAM there on your cognos server ?

I tried to do some google on your issue but did not find any specific link. How about contacting IBM support direct ?

Also one thing I clicked, please check some DB2 related files in your cognos install directory.

Below are steps while connecting to DB2 I got from DQM proven practice guide.

To configure IBM DB2 connectivity for use within IBM Cognos 10 installed on Microsoft Windows, follow these instructions:
Within the IBM DB2 install directory of the database to be used for the connection, locate the ..\SQLIB\JAVA directory.
Within this directory locate and copy the db2jcc4.jar and the db2jcc_license_cu.jar files.
Within the IBM Cognos 10 install directory, locate the ..\v5dataserver\lib and the ..\p2pd\web-inf\lib directory.
Paste the db2jcc4.jar and db2jcc_license_cu.jar files into both these directories.
In order for the IBM DB2 driver to be picked up by IBM Cognos 10, the IBM Cognos 10 service will need to be stopped and started.

[sdf]

Additional to the questions from sunsoft.
Does your server only caters BI or you have other applications running in the server.

To check the cognos service JVM allocation launch cognos config >> under ibm cognos service >> instance name.
You can also monitor your server resources after restarting cognos service.
look into BIBusTKServerMain service.

[Francis aka khayman]

replying because i am interested to know the problem and how it is solved.

while at it, may as well try to contribute... so here goes

is your Cognos server running on Windows or UNIX?

btw, I will also bet on memory or disk space issue as the cause.

[sdf]

'QueryService', 'StartService', 'Success'.
16:28:05, 'com.cognos.pogo.reportservice.ProcessManager', 'pogo', 'Failure'.
DPR-DPR-1035 Dispatcher detected an error.
Accept timed out

how many dispatchers do you have, this might be caused by one of your dispatchers having different password set.

[nsaxena]

Hi All ,

Many thanks for giving valuable suggestions. I checked with the details you all requested.

I would like to point two things first : In our serer,cognos fm is installed in separate directory ( under Program Files x86) and Cognos BI installation is done on C:\Program Files\.. directory.

Comments related to sunsoft
I followed the steps suggested by sunosoft.

I traced that those db jar files are not placed in C:\Program Files\IBM\cognos\c10_64\v5dataserver\lib path.. so i put them. In other path mentioned i had put.

Also do we need to place same in COGNOS PATH where FM is installed (Program Files x86) as it has kinda similar directory structure?

Points asked by sdf
I checkd the T JVM allocation launch cognos config >> under ibm cognos service >> instance name. and find out that it is showin 768 MB. Maximum memory allocation in tomact  is set to 768. Attaching cognos config tool screenshot for the same
Please suggest.

Also we are using only one dispatcher. Anything you want me to check in properties of this? Please guide.

Thanks everyone again for all your suggestion.Waiting for further inputs.

Regards,
Nisha

[sunosoft]

Dont worry about FM install as of now.

So did you try to restart services ?

are you getting the same error ?

[nsaxena]

Hi, no not yet..need to restart.

Will do in a while as some reports are being used by user currently.

Will keep you posted. Thanks!

[cognostechie]

Nisha -

I have seen the same problem and the problem was not the setting or the RAM but the fact that there were
some reports which were very slow and consuming the resources which resulted in other reports having to wait
for a long time. The dispatcher was unable to spawn processes for the reports that were 'waiting' and those
reports eventually timed out. Removing those resource intensive reports (for testing purpose only) resulted
in resolving the problem. If this trial makes your environment proper then you can try to tune those reports
for a better performance.

[nagoole]

Hi nsaxena,

Please share below details:
Cognos installed in single server or distributed  if mutible means How many Content manager server, Dispatcher and Web servers ?

How you restarting the cognos services ?

If you restart order wise after restart did you checked the status in cognos administration ? whether its available or partially available  ?

Please check the disk space where cognos installed on your servers

Thaks,
nagoole

[Francis aka khayman]

i thought after the first sentence you are going to ask for cmplst.txt and cogserver.log  ;D

anyway i think your server run on windows. i am not sure if the concept is similar as a server running in UNIX. but sometimes i kill the BIBus process, that frees up the system

Please share below details:
Cognos installed in single server or distributed  if mutible means How many Content manager server, Dispatcher and Web servers ?

[sdf]

did the same before just to isolate the problem,  while our authors are improving the select statements on some of the reports running.
I don't think this is a good practice since killing the bibus process means forcefully stopping a running report.

[nsaxena]

Hi SK ,

The issue is still there. Cognos is hanged. We are again re-starting services.

Also the .dmp file which is getting created after every re-start,is it normal ?

Dmp file path :- C:\Program Files\IBM\cognos\c10_64\bin64\core.20150915.142659.5660.0001.dmp

Kindly advice ! Thanks!

[MFGF]

Hi SK ,

The issue is still there. Cognos is hanged. We are again re-starting services.

Also the .dmp file which is getting created after every re-start,is it normal ?

Dmp file path :- C:\Program Files\IBM\cognos\c10_64\bin64\core.20150915.142659.5660.0001.dmp

Kindly advice ! Thanks!

Have you logged this with IBM support? Getting advice on a forum is ok for non-critical issues, but this sounds like a big, ugly problem the business can see. I'd log a PMR with IBM if I was you, and see if IBM can diagnose the crash from the dump file.

MF.

[nsaxena]

Hi cognostechie ,

Thanks for sharing your insights.

Our cognos environment is very new and we have around only 25 reports. Most of the reports are using sql as source. Since our datasource is very un symmetric ,we cannot go for fm modelling and then use package for the report source.

Plus the size of database is still small.

[bensah]

Is your problem got resolved? I am facing same kind of issue but it is in solari 5.10 environment and Cognos version is 10.1. Appreciate if you post if you have any updates.

[nsaxena]

Is your problem got resolved? I am facing same kind of issue but it is in solari 5.10 environment and Cognos version is 10.1. Appreciate if you post if you have any updates.

Hi ,
No its still existing :(

[cognostechie]

I can share what I have seen in an environment. Some reports were extremely slow (does not matter on the number of reports you have ) and they were causing the dump file to be created after which the system used to 'hang' . You can check the performance of reports in your environment and its possible that the issue could be something else too.

[MFGF]

Hi ,
No its still existing :(

Did you log it with IBM as I suggested? What are they saying the issue might be?

MF.