Cognos Performance issue

[jmwhitehead]

Hi All,

We currently have a production cognos BI 8.3 installation all on one server (gateway, app tier and cm & oracle 10.2 database for the cm) which is a sun T5120 running Solaris 10. This machine has one UltraSPARC T2 processor which has 8 cores running at 1.2 Ghz with a maximum of 64 simultaneous threads of execution which to application software appear as individual cpus. The machine also has 16GB of main memory and 146GB of disk storage.

The issue we have is that the machine is struggling under any user load, we find that although the system is unresponsive to the report users trying to access it via Cognos Connection the server is barely using any cpu. Often the single java process for the dispatcher shows up as using only between 0-5% of the available cpu leaving the rest of the machine idle.  Is there any way of making this java process use more of the available system resources as at the moment it is struggling to provide ant service at all?

Thanks,

Jonathan

[smiley]

The bitkservermain.exe is single threaded.
Put the processes for interactive reports from 2 to 8, and set the low affinity from 4 to 1.
Then try again.

If still dissapointing, then the report runtime on your 1.2 ghz is the best you can get, and you're better off with cheap wintel dispatchers, running 3.6/3.8 Ghz.

Your baseline is 1 user on the server, running 1 reports.
For the rest cognos specs your servers to have the other user's report runtime to be the same or almost similar.

[jmwhitehead]

Thanks for the reply. I tried these settings and at first they seemed fantastic but over time the system returned to its v.slow and unresponsive state with the presentation service reporting a number of failures and the average response time increasing.

Initially after setting the low affinity to 1 and nunber of interactive processes to 8 we saw a greater number of BItkservermain processes  and then after a while the system ground to an effective halt. We noticed that the number of processes high water mark for reports had maxed out at 8 so we increased the number to 16 and restarted the reports server and then watched the processes running, gradually the number of spawned BItkservermain processes would die of one by one until only one was left and the system again ground to a halt even though there were 30 active user sessions according to the admin page and there were interactive activities listed in a pending status rather than being executed.

We are effectively at a loss as to why this system is so poor, it was at least running albeit slowly around 3 weeks ago, at which time nothing has changed to the best of my knowledge, and it's been downhill ever since.

I contacted IBM cognos support and their advice was to reset all the tuning parameters to default and see what happens, unsurprisingly nothing got better.

Cheers,

Jonathan

[smiley]

8.3 had a memleak which was fixed in SP2.
What are you running?

[jmwhitehead]

its 8.3  with sp4 applied. it used to be sp2 but we upgraded on the advice of ibm support to fp4 to see if that fixed the issue.

[kolonell]

Hi Jonathan

Performance issues on Unix are mostly due to ulimits that are set too low (by default). Can you post the output of the following 2 commands (whilst logged in as the user starting Cognos 8 )
        * ulimit -Sa
        * ulimit -Ha

[jmwhitehead]

Hi,

These are the results from those two commands :-

[cognos@rex ~]$ ulimit -Sa
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 256
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited

[cognos@rex ~]$ ulimit -Ha
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 65536
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited

Curiously after a really bad day yesterday the system is actually performing fairly well today with no major slowdowns yet and the last time it was restarted was 1day and 15 hours ago which at the moment is something of a record.

[jmwhitehead]

A follow up to this issue - IBM support eventually decided that the T5120 and its coolthreads architecture is an unsupported platform for running cognos on so we had to transistion the service onto some zones running on a cluster of sun M series servers instead. So far performance has been a lot better.

[kolonell]

sorry for the late follow up ... holidays ;-)

I would recommend increasing the "open files" and "stack size" (both soft and high ones)