Performance issues in Cognos 8.4.1 BI / Planning

[jeffowentn]

Cognos 8.4.1
FP4 - BI
FP2 - Planning

Mix of Windows 2003 and 2008 servers - all VM's

SQL 2005/2008

Sun Java LDAP / Access Manager 7.3/7.4

12 servers in the mix

Three of our dispatchers are experiencing a high volume of failures for the last few days, and we are not sure why.  To our knowledge, nothing significant has changed in the environment.  This is what we have:

Primary CM server - no issues
Secondary CM server / Secondary BI App server - dispatcher shows roughly 98% failure on the dispatcher, when reviewing under Administration > System
Primary BI App server - dispatcher shows roughly 98% failure on the dispatcher, when reviewing under Administration > System
Primary EP App server - dispatcher shows roughly 90% failure on the dispatcher, when reviewing under Administration > System
Secondary EP App server - no issues
3 EP Job servers - no issues
2 Planning Web servers - no issues
2 Gateway servers - no issues that we can tell, but there are no dispatchers running on these boxes

1.  No recent updates to cogserver.log on any of the servers with issues.
2.  No Windows errors that I can see in the Event Manager for any of the servers.
3.  All three affected servers plus the Primary CM server and the Secondary EP App server have received the same Windows security updates on 8/19/13
4.  The two BI App servers with issues showing appear to be specific to the Presentation Service - only enabled on these two servers - not on the Primary EP App server
5.  The Primary EP App server is also where Sun Java LDAP and Access Manager are running to manage the security in the environment
6.  While there is not a lot of activity going on, with our users (between cycles), there are no users complaining (however, I want to be proactive and keep from having issues in a few weeks when the cycle ramps up, again; and, the failure rates are typically more along the lines of 10-40%, instead of 90%+)

Any suggestions on what else I should look at and how I can better troubleshoot this issue?

On a side note, we have a Foundry load balancer appliance in front of the two gateway servers.  We are currently in the process of swapping that out with an F5 load balancer appliance, but we have not cut-over, yet.  We've created the new VIP so we can point to it, directly, and test the new appliance, but we have not changed the DNS entries to point to the new VIP.  They are still pointing to the old VIP.  Any chance the new appliance is causing all these errors when it is only testing that the two gateway servers are up and running and that a specific string is presented by the servers?  The load balancer has no ability to log into Cognos Connection or do anything else in the environment.

Any help or insight you can offer would be greatly appreciated.  I've tried to put enough information in here to be helpful without putting way too much (maybe).  Let me know what else you need to know.

Jeff

[Grim]

I had something similar earlier this year with my 8.4.1 install, FP5 resolved some of the issues, but we also had a bad user running a nasty report that was pulling 80+ gigs of information in then doing local processing (she was using cognos as an etl tool -.- ) causing the Dispatcher Java's to run into mem overflow and core. Since killing this report and FP5 no more issues of this nature.

[jeffowentn]

Grim,

Thank you for the quick response.  When testing F5, we were actually seeing some latency in the performance (pages taking a bit longer to load, and oddly enough the past activities had problems ever loading in Administration).

As for reports, we are have pretty much been reduced to a Cognos Planning shop, as most all of the BI content has been migrated to SAP BOBJ/BW.  I do not believe any new content has been created in Cognos BI, but I will check and find out...

Jeff

[jeffowentn]

Do you expect FP5 to resolve this issue for Planning?

[Grim]

Do you expect FP5 to resolve this issue for Planning?

Can't say. I've never been in a shop that runs Planning, nor have I supported Planning. I'll have to defer to the Planning Gurus that lurk here. ;o)