Need help For Perfomance optimization

[richiearora]

Hi All,

I am working on a Perfomance optimization for my Project on the Cognos Environment.

Problem Statement: Load Balancing for the Enviroment with 8 Dispatchers.

Variables:

We have 2 Business critical reports(One Package) + 60 Cognos Jobs(Both Querying Diffrent Databases and Runs 21 Hours Daily(Both Seperate packages))

Currently our environement is Experiencing Lot of Failures, hence we have Decided to Add Additional 6 Dispatchers on the Envi.

Now Considering the Requirement can you guys suggest me how do i do the Advance Routing on the Envi.

Straight Forward Scenario:
2 Dispatchers : priortize Reports
2 Dispatcher for One Job Each and Rest 4 Dispatchers for rest 26 Reports.

Please help.....

[bvk.cognoise]

Hi,
please follow the steps it may helps you.

The process of tuning and optimizing IBM Cognos 8 environment may looking at the following settings:
>Set usage peak periods - for instance set the number of connections and processes to low during peak hours (business operation hours)
>Adjust the maximum number of processes and connections for the report service, the batch report service and the data movement service
>Adjust queue time limits for interactive requests
>Adjust PDF output settings - file character encoding, font embedding and compression policy
>Maximum execution time for the report service, batch report service and data movement service
>Adjust retention time for watch list report output (for Analysis Studio and Report Studio)
>Change compression for email attachments
>Adjust the report size limit for the report data service
>Server group names for advanced dispatcher routing
>Load balancing - balancing requests among dispatchers by adjusting weights
>Within a load balancing infrastructure, use a cluster compatible mode for dispatchers (load balancing)

thanks
bvk

[RobsWalker68]

Hi,

I'm not sure if adding extra dispatchers to your system is necessarily going to solve your problem.

Personally, I would add extra dispatchers but only after I have analysed what is going on in the system and on my databases and identify where the bottlenecks sit within the system.  It will be easier to advise you if you can tell us what is causing the issues/failures as this will determine how you allocate services on the extra dispatchers and set up advanced routing etc

We have 2 Business critical reports(One Package) + 60 Cognos Jobs(Both Querying Diffrent Databases and Runs 21 Hours Daily(Both Seperate packages))

Is this all of the activity on the system or do you have in addition a very large user base that works interactively on the system?  On the face of it what is running doesn't look large  and something that requires 8 dispatchers.

Out of interest I'm curious to know why 8 dispatchers why not 3,4,5 and build up incrementally.  What was the reasoning behind the 8 dispatcher choice?

Kind Regards


Rob

[richiearora]

Hi Rob,

thanks for the reply, Appricate the Same.

First thing, we got Approvals for Adding 6 new Dispatchers and Suggested the Number "6" by the IBM Sales for pur Application team.

We have our Envi Like this Currently:

2 Dispatcher(Only App Component) - 32 GB ram each

We are Adding 6 New Dispatchers (3 Dispatchers on one App Comp and Same on other)

So basically we have 6 Approved Dispatchers and we would Like to Add that to the Envi.

Our environment have 1000 Users, For One Month Stats of our Envi:

11600 reports got executed out of which 5371(Were the Business Critical reports)

Total report Failure is 559.

So to answer your question, Apart from the Jobs i have discussed previously more 50% Execution will still be there on Envi.

We were thinking of making new packages for each Job(Which runs 21 Hous Daily) and Executes 5000 Reports everyday.
And Package level routing for this Jobs on 2 Dispatchers respectively.

Rest 4 Dispatchers for 2 Business Critical reports and Rest 2 Disp for Environemtn(Interactive reports).

Please comment.

Thanks In Advance

[RobsWalker68]

Hi richiearora,

Obvious question first.  If IBM are recommending 6 new dispatchers and I presume they have looked at your system metrics first, what do they recommend?  They have identified the bottlneck and therefore the  need for extra dispatchers so I guess they should be able to profer some advice to you.  I think you should at least make them work for your maintenance renewal  ;)   

As you will appreciate without really having a feel of your system it is hard to tell you how to allocate it and you obviously have some ideas already how you are going to set it up.

I think how I would approach it ,and I'm no expert on this, is to examine my queue metrics just to confirm that my system is not keeping up with demand and therefore indicate I perhaps need extra resources.

Then I would look at the request metrics for each of the services and see what services have the highest failure rate.  Is it report service, batch report service etc.  If I was getting high failure rates on a particular service then I would do some investigation and try to understand what the root cause is. The system metrics may show that your Cognos system and its throughput is ok but your database is labouring or perhaps its a handful of complex reports that need a re-design.

If I identified resource issues with a given service(s) then that would then lead me down the road of allocating the extra resources to it. For example if I had a very intensive batch reporting/scheduled environment then I might dedicate a dispatcher to batch reporting only and disable the other services iI don't need.

Perhaps something to consider is your Content Manager.  If a lot of your output is saved output like PDF/Excel etc then it may be worthwhile considering a dedicated CM server and making sure this is a decent spec i.e don't put the CM on the slowest server you can find. 

Rest 4 Dispatchers for 2 Business Critical reports and Rest 2 Disp for Environemtn(Interactive reports).

I'm not sure if dedicating 4 dispatchers to 2 business critical reports will help as only 1 disptacher will be running a report at any given time.  2 dispatchers for the interactive reports doesn't sound unreasonable as far as I can tell.

Like I say I'm no expert on this but hope some it helps

Kind Regards


Rob
   

[richiearora]

Hi rob,

Thanks for your Input.

Really appreciate the option of Dedicated CM, Havent thought on that prespective.

Out of Curosoty would Like to ask,

"I'm not sure if dedicating 4 dispatchers to 2 business critical reports will help as only 1 disptacher will be running a report at any given time.  2 dispatchers for the interactive reports doesn't sound unreasonable as far as I can tell.
"

Can you explain me a little bit about the above Statement.

Thanks,
Richie

[RobsWalker68]

Rest 4 Dispatchers for 2 Business Critical reports and Rest 2 Disp for Environemtn(Interactive reports).

Re-reading it I may have misunderstood your post.

Are you saying 4 dispatchers(app tier) in total ,2 of which to the critical reports and the other 2 for intereactive reports.  Not 4 dispatchers dedicated to the business critical reports

Rgds

Rob

[richiearora]

Yes...

I mean in total 4 Dispatchers, 2 for Business Critical and 2 for Interactive reports.

[RobsWalker68]

Ok,

That makes a bit more sense.  I guess you could set up a server group for the 2 dispatchers handling the business critical reports and then use advanced dispatcher routing at the package level to direct to that group.

Cheers

Rob

[richiearora]

Thanks Rob,

Will do a POC for same, any idea on how to set the Process Capacity and Affinity connections.

And do i have to go to Weighted Round Robin or Clustered?

Richie

[RobsWalker68]

Hi Richie,

Have a look at the documents below re tuning of the processes.

IBM Performance Tuning Settings
http://www.ibm.com/developerworks/data/library/cognos/page62.html

Performance Tuning Cheat Sheet
http://www.ibm.com/developerworks/data/library/cognos/page61.html

You should set Weighted Round Robin unless you are using external routers to do the load balancing then you should use Clustered.

Kind Regards

Rob

[richiearora]

Thanks Rob for the Documents,

Update: We have Tried for a Advanc Routing options as Discussed Earlier.


We have total of 4 quad Core Processors

We have allocated 4 Dispatchers for 2 Business Critical Report and 4 Dispatcher for Jobs, and Routing on Package.

POC: This havent shown us a Significant Performance Gain.

:(

Please help with the Solution.

We are thinking of various Options No

Affinity High : 1
Low Affinity : 4

[RobsWalker68]

Hi Ritchie,

Possibly as mentioned before this may not be an issue with your architecture.  Are you sure you have an issue with the resources (i.e. the ability of the Cognos architecture to process the requests in a timely manner) or do you have an issue with certain long running reports?

If you have a lot of requests being received and therefore you are finding a lot of reports are queuing (analyse queue metrics) then this may indicate that there is not sufficient processing power to process the number of reports and adding hardware would have potentially helped.  If this is not the case then adding extra capacity will just lead to spare capacity on the system without much overall benefit.

Is it all reports on the system that are taking a long time to run or a few i.e. is the system overall generally sluggish?  Have you used the audit database to identify those long running reports so you know what to concentrate on.

If it is a small group of business critical reports then perhaps look at redesigning or re-architecting them.  For instance I recently completed a project where the existing relationally driven reports took an hour to complete but when transferred to a cube based solution they took 15 seconds.

Have you checked with the DBA so the queries can be analysed.  It may be a database issue.

It is possible that if you have a complex system then its a bit of anything so the best approach is to get all your facts and figures together looking at the Cognos system metrics to identify any resource issues, identify long running reports for further analysis/design  and supporting database information.

Unfortunately as your most probably finding this is as much art as it is exact science and the only way to solve it is break it down, gather the facts and let these lead you to the solution. 

As the the fictional character Sherlock Holmes said "when you have eliminated all else, whatever remains - no matter how unlikely - must be the truth"  ;D  Of course as we are not Sherlock Holmes, IBM have kindly provided a system management methodology to help with this type of issue and the link is below.
http://www.ibm.com/developerworks/data/library/cognos/page258.html

Hope this helps

Kind Regards


Rob 






 

[richiearora]

Hi Rob,

thanks for the Response again, I will Assess the Same.

Let me Breif you a Little, we have total of 30 reports and in those 2 reports are very critical and takes 50% of Load of our Envi.

Now going forward we will be adding 2 More Jobs(Which are 60 Jobs ) to the Envi.

So the Resource upgrade is something which is mandatory.

Now the Problem is how do we Load Balance these Things.

1: Make 3 Server Groups - 1 Group with 2 Dispatcher 2nd Group with 3 Dispatcher and Rest as Third Group.

We Have total 3 Packages for whole Envi- 1 For 28 reports , 2 For Business Critical reports and 1 For Jobs.

We have done a Advance rouyte based on package, and See no good result as i mentioned.

These 8 Dispatchers are on 2 Dual Core Physical and 2 Quad Core Processer and No of CPU is 2 2 4 4 for Physical machines.

Questions:

1. When we do a Package Level route, Does that mean If i Configure a Package with a Particular Server Lest Say A, it will Execute with ony that Server and FAILS if doesnt find Processes on that particulat Server

OR

My understanding, The Advance Route will Just priortize based on Routing, where as the Request will still process if the other Server Groups are not Busy...

2: Now Cognos Support have Come up a Solution for Jobs they want to route on a group rather than a Package.

Can help how will this help, if you have any input?

3.

The Affinity connections they have asked us to Change to

High - 2
Low - 4

Previously

High - 1
Low- 4

Can you heklp me with good Article which helps to understand this?

Appreciate the Patience,
Richie

[RobsWalker68]

Hi Ritchie,

You say

2 reports are very critical and takes 50% of Load of our Envi.

Can you elaborate on what 50% load of the environment means as it may help people advise you further.

When we do a Package Level route, Does that mean If i Configure a Package with a Particular Server Lest Say A, it will Execute with ony that Server and FAILS if doesnt find Processes on that particulat Server

Forgetting about the presentation service/content manager flow elements.

From my understanding the web gateway configuration will potentially have a list of multiple dispatchers with the first dispatcher in the list designated as the primary dispatcher, the rest as failover.  On receiving a request from the client the web gateway will pass this to the primary dispatcher and if this is part of your dispatcher routing group it will potentially process it, otherwise if not part of the routing it will pass this onto the designated dispatcher/server group.  This will be placed in the queue for processing and depending on the resources it will suceed or fail (queue timeout etc).

With regards to the affinity connections the IBM training guide recommends the sum of the high and low affinity connections should not exceed 5, although if this is coming from IBM support I would go with their recommnedations and at least try it out.

You may want to also try asking on the IBM developerworks forum below and especially Phil W as he is pretty good at these type of questions.

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1573

Cheers

Rob

[richiearora]

What i mean by 50% of load is

My reports Execution is 1000 Reports Per day, In that 500 Reports are the 2 Business Critical Reports.

I have Posted in Dev works also.

Thanks

[ykud]

How did the story end up?

I had a similar excercise this summer and it's crucial to monitor physical server load and increase number of Report Services (that'll generate more BIBus processes and even the load a bit). Errors are usually triggered when a single BIBus gets more request than it can handle and dies under the load. Having more BIBuses means less errors and, from some point, slower server reponse, since they're racing for the same physical resource.