EP 8.4 - Capacity

[jonathan.webb@raytheon.co]

Hi everyone - hoping someone can advise. I'm new to EP but experienced in BI.

We are using Cognos EP 8.4 and are having really slow performance using Contributor Web at peak periods - approx 70 users simultaneously accessing Contributor. We've got a VM server dedicated for the Planning Web and Planning Data services with 4 CPUs / 4GB RAM and we're going to clone this to double the capacity.

Can anyone suggest the expected capacity of a server with this spec.. trying to give a guideline on how big our architecture should be scaled to.

Thanks in advance
Jon

[Rutulian]

Hi Jon,

Planning Web Grid access should be fairly comfortable up to 100ish users on a 2thread/2cpu box if sharing with other low-need services like presentation service or a gateway.  I'd suggest looking at what else is running on the box at that time.

The thing that caught my eye here was the Planning Data Service - are you running live reports from planning data?  If so I would very strongly encourage you to look into Incremental Publish, if you're aggregating across the e.list you might find that a report run off 'live' EP data can take 20 minutes to run while one run off a 5-minutely incremental publish comes back with data that was live 5 minutes ago and takes 2 minutes to run.  That actually gives you a more up to date report.

I would avoid the use of PDS unless you have some very strange requirements.  Back in 8.1 it was the only option, but now we have Incremental Publish it's a better solution in most situations I've come across.

[jonathan.webb@raytheon.co]

Hi - thanks for your reply.

We don't sit Cognos BI directly on top of Contributor as a data source, however I have built a datamart and use incremental table publishes to populate that for reporting.

We've got a 3 server architecture.. the gateway/content manager on one, the Planning Web & Data services on another (this one), and finally the admin console & job service on a third.

Is the planning data service a mandatory service to be on - what do you think we might lose if we disable it?

Appreciate the advice

[Rutulian]

Hi Jon,

Sounds like you're using best practices there, that's the way I'd split across 3 boxes too and you're doing the incrementals properly... nothing much else running on the Web box, which with a spec like that I would have thought should be capable of dealing with your users.

Planning Data Service isn't mandatory, if you turn it off all that will happen is you won't be able to use the packages created at GTP to build reports.  As doing so would not be a good idea anyway and you have an incrementally fed datamart, this shouldn't hurt.

I'd consider monitoring your database for unwanted locking behaviour as well as checking CPU utilisation on the Web/PDS and Gateway/CM boxes during peak times.  Contributor saves can lock items higher than them in the e.list hierarchy as rollups are processed, I wonder if you could be seeing problems due to that.

There are also general performance tips like avoid multi e.list items and make use of access tables that I think you're probably on top of.

I wonder if anyone on here can comment about tricks like dummy hierarchy items or DB row-vs-table-locking configuration tweaks for similar situations?

Or even about how well the Contributor Web Service can spawn threads to handle requests?  Not looking for an official answer, just whether distributing to let round robin do its thing is likely to help.

Interesting problem - thanks.

[ericlfg]

Hey Jon,

You're in good hands with Rutulian, and he's already covered a lot of the stuff I would have looked into.  To be more specific, what areas seem to be sluggish during heavy load?  Saves?  Switching between tabs?  Calculations (Entering data and hitting enter), etc?

Since it seems to be when there is heavy load by the end users, Rutulian has already advised monitoring CPU usage and DB aspects.  I'd also want to confirm:

1.  How often your incremental publishes are run and if any administrative jobs (data loads, admin link execution, reconciles, etc) are run during the day along with the end users utilizing the environment. 

2. How many VMs are run on the background architecture that this VM machine is running on. 

3.  Using cognos.cgi or cognosispai.dll?

As far as I know, the Web service is single threaded and runs within the epPlanningService8.exe - It's designed for fail over in the case that another system running the service becomes unavailable.

Cheers

[jonathan.webb@raytheon.co]

Hi both,

Thanks for the advice - really helpful.

Our content store and planning data store (and all publish containers) are stored in a single SQL 2000 environment. Although it is busy, I think it is rarely flat out and don't think it is contributing to the issues.

The planning web / data server runs more or less at 100% over 4 processors during the peak periods so definitely seems to be a capacity issue here.

The incremental publishes are being run every 2 hours during the day in peak periods, followed by a downstream ETL process in SQL to pull that data into a datamart and powercube for reporting. Again, I didn't think this was particularly intensive work as I'm guessing the publish is run from the job server which is on a separate VM instance.

I believe our VM servers are set up in a decent way.. one of my first suspicions was that the background architecture was getting hammered by something and our VM was suffering. Don't think that's the case though.

We're using the ISAPI interface (everything built on Windows/IIS) so no issues there presumably.

We're going to chuck in another VM box to run the Planning Web Service only and use the round robin to load balance between the 2 boxes... hopefully that will resolve the issues. I'm also thinking that this gives us a good excuse to go for C10 as there's meant to be a whole bunch of performance improvements delivered by the upgrade.

If it happens again I'll get out network admins and DBAs to monitor the environment more closely.

Thanks for the advice

[ericlfg]

Hi Jon,

I wouldn't expect that your second system identified, the Planning Web & Data services machine, to be running as hard as it is.  However I suppose it is possible. 

I do not believe the PWS service load balances in the way regular dispatchers would handle requests (open to being corrected).  I believe it's completely internal to the service and the premise is that it's going to share the cache of what's been requested between all machines running the service.

Ex: ServerA and ServerB are both configured for the PWS.  When an end user requests a model that hasn't been cached yet the services and background code will load the model into the cache.  However, it's not the PWS that services the requests directly.

It is not likely a huge challenge to add another VM to this environment for the purposes of running the PWS, and I'd be very interested to know how you make out following this configuration test.