Query Service CPU Usage

[dougp]

Cognos 11.0.13 on Windows Server 2016.

The Report service is configurable so that I can specify how many reports can run in one session (Number of low affinity connections for the report service during peak hours) and how many processes Cognos can start (Maximum number of processes for the report service during peak hours).  So in Task Manager, I'll see many instances of BIBusTKServerMain.exe.  If I'm having trouble with a runaway report (maybe no filters) or the filter condition dialog bug, I can kill the process that is using 100% of one CPU core and users can continue to work.

We're currently conducting user training for a new data mart.  The model is DQM.  For some reason, the CPU usage is currently 100%, and it's all

<install_location>\jre\bin\java.exe -Xms...

It's using nearly 100% of all CPUs on the system (multipe cores), so it must be multi-threaded.

I think that's the Query service.  Is that correct?

I don't see the same configuration options for the Query service.  Do all DQM reports run in a single java.exe process?  Is there any way to identify which report or user is causing the high CPU usage?

[MFGF]

Cognos 11.0.13 on Windows Server 2016.

The Report service is configurable so that I can specify how many reports can run in one session (Number of low affinity connections for the report service during peak hours) and how many processes Cognos can start (Maximum number of processes for the report service during peak hours).  So in Task Manager, I'll see many instances of BIBusTKServerMain.exe.  If I'm having trouble with a runaway report (maybe no filters) or the filter condition dialog bug, I can kill the process that is using 100% of one CPU core and users can continue to work.

We're currently conducting user training for a new data mart.  The model is DQM.  For some reason, the CPU usage is currently 100%, and it's all

<install_location>\jre\bin\java.exe -Xms...

It's using nearly 100% of all CPUs on the system (multipe cores), so it must be multi-threaded.

I think that's the Query service.  Is that correct?

I don't see the same configuration options for the Query service.  Do all DQM reports run in a single java.exe process?  Is there any way to identify which report or user is causing the high CPU usage?

Hi Doug,

It could well be the Query Service. Can you see what its parent process ID is?

How is the memory pool for the query service currently configured? Have you allocated enough memory to it?

Cheers!

MF.

[dougp]

I don't see a way to identify the parent process.  Although, I think this is the top-level process.

I think this is the Maximum memory for Websphere Liberty Profile in MB setting.  It's 8192.  It seems to be more than enough.

I've not had CPU problems exactly like this before.  CPU usage randomly jumped to 100% at 10:30 this morning and stayed there.  I'm not concerned about my config, I just want to know how to identify the specific cause of the problem so I can respond appropriately.  With CQM it's easy because the Audit database tells me which report runs have started and not completed and are using the specific Windows process ID.  I don't see a way to do the same with DQM.  It looks like it's all or nothing.

Would stopping and starting the Query service from within Cognos Administration have been a good solution?  It seems it would be less disruptive than restarting everything.

I have seen BIBusTKServerMain.exe using 100% of one CPU core due to a bug in the Filter Condition dialog.  That's because BIBusTKServerMain.exe is confined to one core.  Could the same bug affect the Query service (java.exe) the same way (and use 100% of all cores) if the package being used is using DQM?  Yet another argument for 11.1.7.

[MFGF]

I don't see a way to identify the parent process.  Although, I think this is the top-level process.

I think this is the Maximum memory for Websphere Liberty Profile in MB setting.  It's 8192.  It seems to be more than enough.

I've not had CPU problems exactly like this before.  CPU usage randomly jumped to 100% at 10:30 this morning and stayed there.  I'm not concerned about my config, I just want to know how to identify the specific cause of the problem so I can respond appropriately.  With CQM it's easy because the Audit database tells me which report runs have started and not completed and are using the specific Windows process ID.  I don't see a way to do the same with DQM.  It looks like it's all or nothing.

Would stopping and starting the Query service from within Cognos Administration have been a good solution?  It seems it would be less disruptive than restarting everything.

I have seen BIBusTKServerMain.exe using 100% of one CPU core due to a bug in the Filter Condition dialog.  That's because BIBusTKServerMain.exe is confined to one core.  Could the same bug affect the Query service (java.exe) the same way (and use 100% of all cores) if the package being used is using DQM?  Yet another argument for 11.1.7.

Hi Doug,

The Maximum memory for Websphere Liberty Profile in MB setting isn't the one for the Query Service - that's for the main servlet container for CA, and runs as a java process in its own right. You'll need to go into the Admin Console and into the properties of the Query Service to see the initial and maximum heap size settings, and change them. If you haven't touched these, they will both be set to 1Gb, which isn't enough. The bare minimum IBM recommends is a 4Gb initial heap size and an 8Gb maximum heap size. Depending on usage, you may need to tweak these higher too - if you have enough free memory on your Cognos server(s) to accommodate. Here's the official IBM recommendation:

https://www.ibm.com/support/pages/node/293851

I have seen the CPU go wild like this in the past where the Query Service container was too small. I was using Dynamic Cubes at the time, which is a whole other story in terms of Query Service memory sizing. As a first step, re-configure the Query Service memory settings and see if that fixes things.

For the question about the BIBusTKServerMain.exe bug, no, it's not something you see in DQM. The Query Service is 64-bit and runs as java servlet processes (within a java.exe container) across multiple cores.

Cheers!

MF.

[dougp]

Well, DQM doesn't generally cause problems like that for me.  My max heap size was already set to 8GB.  Since I have plenty of RAM, I increased the values to 4GB/16GB.  So, thanks for that.

This only happened once and I am unable to reproduce it.

I didn't notice high memory usage for the Query service this time, just high CPU usage. 

I have seen wild Query service CPU usage from one other thing:  When validating a model, FM changed the data type for every data source query subject to "unsupported".  Validation reported everything was OK, so I moved on.  Oddly, the model was using CQM, but the Query service always wanted 1GB more RAM than it had available.  (If it was configured for 16GB, Task Manager would show usage grow to 17GB, then the 100% CPU usage would start.)  The fix was to validate the model again, spot-check data types, and republish the affected packages.