Cognos Errors

[Kavya]

Hi All,

I need help in resolving few of the Cognos errors that I am seeing on my new installation. I did install Cognos 10.2.1 on Linux servers. We are on three tier structure. When I restart, all 3 dispatchers starts successfully and show available. But, after few hours only the primary one show up as  available and other two show up as Unknown status and along with that I see three other errors:
DPR-ERR-2058 The dispatcher encountered an error while servicing a request: Process 'error.def' failed to execute.
PRS-MSG-0558 The message file "messages/cogadminuimsgs.xml" is missing.
PRS-INS-1420 The "XSLT" instruction could not build the full path from the base path "../template/ps" and "/portal/config_rt_ui.xslt".

I have only two auto reports scheduled to run every hour on SCI. I do get those emails every hour. But when I login to SCI, I see these errors. There is no one accessing SCI. All the activity on this new built sci is only two auto email reports running every hour.  Attached is the picture of the errors I see.

Can someone please help me in resolving these errors. Please let us know if you need any additional information from me.

[misscognos]

I can't see the attachment.. but it sounds like:

1. Could just be a bad/damaged install.
2. Did you install as one Unix user and start as a different user? In other words Unix permissions.

[Kavya]

I did install using the same user on all servers.
Some times, I see PRS - TRS - 0909, PRS - TRS - 0902, PRS - INS -1425 errors.

I am getting the CPU Maximum usage alert frequently on these servers. When I get these, sometimes I get to login to SCI and sometimes I do not get the login page at all. CPU alert goes down in few minutes but still the SCI does not give the login page.
When I get to login, I see those above errors or only one dispatcher shows as available and other two shows up as Unknown or sometimes I get Gateway 504 error when I am trying to look at the System Status.
But, when I look at the past activities, reports ran successfully against these unknown servers.
Also, the dispatcher URL (http://servername:9300/p2pd/servlet/dispatch) shows login page and I am able to login there without any errors.

Couldn't find out where could be the problem.

[Kavya]

Can anyone please help me understand what causes the dispatchers to hit high CPU usage often and how to resolve that?

[MFGF]

Hi,

Are you seeing anything of interest recorded in cogserver.log? From the symptoms you describe, I'm wondering whether you have enough memory allocated to the main java heap, and/or the query service? What are your current memory settings for these?

MF.

[Kavya]

Hi,

The dispatchers memory is 7.68 GB
Maximum Memory for Tomcat in MB is set to 768 in IBM cognos Configuration. ( I believe this is the same for JVM)
If there is anything else for me to check, then could you please give me the Linux commands to run.

I did check the cogserver log and it shows some errors like:
CM-REQ-2378 A request was received to start the negotiation process to determine which Content Manager instance should become active.

DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Failed to send HTTP request or read HTTP response</messageString></message></messages>   DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node. For more information, see the dispatcher and Content Manager detailed logs. Ensure that the local Content Manager service is started.

Sorry, I am not sure how to resolve these errors looking at the logs. Could you please help me with understanding these logs and the problems here. Please excuse me for my lack of knowledge on all these as this is first time I am installing cognos on Linux machine.

[MFGF]

Hi,

The dispatchers memory is 7.68 GB
Maximum Memory for Tomcat in MB is set to 768 in IBM cognos Configuration. ( I believe this is the same for JVM)
If there is anything else for me to check, then could you please give me the Linux commands to run.

I did check the cogserver log and it shows some errors like:
CM-REQ-2378 A request was received to start the negotiation process to determine which Content Manager instance should become active.

DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Failed to send HTTP request or read HTTP response</messageString></message></messages>   DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node. For more information, see the dispatcher and Content Manager detailed logs. Ensure that the local Content Manager service is started.

Sorry, I am not sure how to resolve these errors looking at the logs. Could you please help me with understanding these logs and the problems here. Please excuse me for my lack of knowledge on all these as this is first time I am installing cognos on Linux machine.

Hi,

What are the min and max settings for the JVM Heap Size for the Query Service? You can find the Query Service in Admin Console > System then go into its properties to see.

The DPR-DPR-1035 errors are a good indication that you have something amiss in your configuration. Here are some tech notes of possible issues to investigate:

https://www-01.ibm.com/support/docview.wss?uid=swg21579053
https://www-01.ibm.com/support/docview.wss?uid=swg21338756
https://www-01.ibm.com/support/docview.wss?uid=swg21392605
https://www-01.ibm.com/support/docview.wss?uid=swg21502745

Cheers!

MF.

[Kavya]

Thanks MF for the reply.

On all the three dispatchers, I see that
Initial JVM heap size for the query service is set to 1024 MB
and JVM heap size limit for the query service is set to 1024 MB

I found that limit conf file was missing on the web servers. I have added that to webservers ( we have 2 web servers and 5 application servers (or dispatchers) for this SCI). I have this in this file:

wm  soft  fsize   unlimited
wm  hard  fsize   unlimited
wm  soft  core    unlimited
wm  hard  core    unlimited
wm  soft  cpu     unlimited
wm  hard  cpu     unlimited
wm  soft  data    unlimited
wm  hard  data    unlimited
wm  soft  stack   unlimited
wm  hard  stack   unlimited
wm  soft  rss     2097151
wm  hard  rss     2097151
wm  soft  nofile  90000
wm  hard  nofile  90000
wm  soft  nproc   96049
wm  hard  nproc   96049

After adding this file, I have restarted all three dispatchers and even after that i was getting high CPU usage alerts on second and third dispatchers but not on first one. As stated, in this below article, I stopped all three dispatchers, followed the steps mentioned in this article and the restarted my first dispatcher.

https://www-01.ibm.com/support/docview.wss?uid=swg21963616

I did not see any single error for 3 days. on the fourth day, I have restarted the second dispatcher. Just after few hours, this second one hitting up high cpu usage. And these are few error messages I am seeing in the cogserver log:

DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Failed to send HTTP request or read HTTP response</messageString></message></messages>   
DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node
CM-REQ-2378 A request was received to start the negotiation process to determine which Content Manager instance should become active.
Original Error: DPR-DPR-1034 Unable to complete the request. The query to retrieve the 'canUseServerAdministrationTool' capability failed.
Content Manager is unable to connect to the content store. Verify that the database connection properties in the configuration tool are correct and that when you test the connection, the test is successful. Could not commit with auto-commit set on</message></messages>

I did check the content DB connection in the cognos configuration file and it does show as successful.

Any Suggestion please? Does reinstalling cognos on second and third dispatchers will be a good idea?

[MFGF]

Thanks MF for the reply.

On all the three dispatchers, I see that
Initial JVM heap size for the query service is set to 1024 MB
and JVM heap size limit for the query service is set to 1024 MB

I found that limit conf file was missing on the web servers. I have added that to webservers ( we have 2 web servers and 5 application servers (or dispatchers) for this SCI). I have this in this file:

wm  soft  fsize   unlimited
wm  hard  fsize   unlimited
wm  soft  core    unlimited
wm  hard  core    unlimited
wm  soft  cpu     unlimited
wm  hard  cpu     unlimited
wm  soft  data    unlimited
wm  hard  data    unlimited
wm  soft  stack   unlimited
wm  hard  stack   unlimited
wm  soft  rss     2097151
wm  hard  rss     2097151
wm  soft  nofile  90000
wm  hard  nofile  90000
wm  soft  nproc   96049
wm  hard  nproc   96049

After adding this file, I have restarted all three dispatchers and even after that i was getting high CPU usage alerts on second and third dispatchers but not on first one. As stated, in this below article, I stopped all three dispatchers, followed the steps mentioned in this article and the restarted my first dispatcher.

https://www-01.ibm.com/support/docview.wss?uid=swg21963616

I did not see any single error for 3 days. on the fourth day, I have restarted the second dispatcher. Just after few hours, this second one hitting up high cpu usage. And these are few error messages I am seeing in the cogserver log:

DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Failed to send HTTP request or read HTTP response</messageString></message></messages>   
DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node
CM-REQ-2378 A request was received to start the negotiation process to determine which Content Manager instance should become active.
Original Error: DPR-DPR-1034 Unable to complete the request. The query to retrieve the 'canUseServerAdministrationTool' capability failed.
Content Manager is unable to connect to the content store. Verify that the database connection properties in the configuration tool are correct and that when you test the connection, the test is successful. Could not commit with auto-commit set on</message></messages>

I did check the content DB connection in the cognos configuration file and it does show as successful.

Any Suggestion please? Does reinstalling cognos on second and third dispatchers will be a good idea?

Hi,

You're running on Linux, right? I found an interesting technote mentioning having ulimit set too low might cause this issue:

https://www-01.ibm.com/support/docview.wss?uid=swg21686229

I also found these, which may or may not be of help:

https://www-01.ibm.com/support/docview.wss?uid=swg21963616
https://www-01.ibm.com/support/docview.wss?uid=swg21339758

On the topic of the Query Service memory settings, are you using packages published in Dynamic Query Mode? If so, upping the size limit to 8Gb might be something to try? How much spare memory capacity does your server have?

MF.

[Kavya]

Thanks MF for the reply.

Yes, SCI is built on Linux servers. I will check these articles that you have shared.
I am not sure, if I have mentioned this earlier in my posts, just wanted to give one more detail.
As soon as I get the high cpu usage alert on the dispatcher, I also receive a notification that CAM_LPSvr, Cogbootstrap, JAVA is in fault state. But, I would be able to login to: http://servername:9300/p2pd/servlet/dispatch and I also see the scheduled report running successfully sometime and failing sometime on this server.

[Kavya]

The limits Conf file was missing in the dispatchers as well. Earlier, I have added this to the webservers only. I have added this same file to the app servers and after that I did not see any high cpu usage alerts or any dispatchers errors. SCI is running stable since last 6 days after adding this limits conf file to the app servers.