11.0.6 Query Service Failure to start

[gohabsgo]

Hey Folks,
We've begun our 11.0.6 upgrade investigation and have run into an odd issue.  We first installed and configured everything without the optional gateway, got all the audit packages and samples up and running and everything was working great.  We then configured IIS and even that was working really well.  Our instance was cycled after that and now when starting the service the Query Service fails to start.  Everything else comes up successfully and we can navigate through the portal without issue (via SG or the IIS gw) but when we go to run a report or test a data source it fails with the following error;

DPR-ERR-2002 Unable to execute the request because there were no connections to the process available within the configured time limit.

Taking a look through the cogserver reveals the following information;

QueryService failed to start with the following command-line arguments: [C:\Program Files\ibm\cognos11_0_6\analytics\jre\bin\java.exe, -Xms1024m, -Xmx8192m, -Xmnx4096m, -Xmns512m, -server, -Xjit:disableLoopVersioner, -Xscmx100m, -Xshareclasses:cachedir=../javasharedresources,name=cognos10%u,nonfatal, -Xmso512K, -Xcompressedrefs, -verbose:gc, -Xverbosegclog:C:\Program Files\ibm\cognos11_0_6\analytics\logs\XQE\dq_verbosegc_%Y%m%d.%H%M%S.%pid.log,1,1000, -Xgcpolicy:gencon, -Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec=cmd.exe /c del /f /s \"C:\Program Files\ibm\cognos11_0_6\analytics\data\.xqe.%pid\" > nul, -DCOG_ROOT=C:/Program Files/ibm/cognos11_0_6/analytics, -DParentProcessID=8448, -Dcubingservices_home=C:/Program Files/ibm/cognos11_0_6/analytics/v5dataserver, -Dcubingservices_data=C:\Program Files\ibm\cognos11_0_6\analytics\data, -DParentProcessPort=49365, -Djava.library.path=C:/Program Files/ibm/cognos11_0_6/analytics/bin64;, -DenableTraceServer, -ea, -Dorg.dom4j.QName.singleton.strategy=org.dom4j.util.PerThreadSingleton, -Dorg.dom4j.DocumentFactory.singleton.strategy=org.dom4j.util.PerThreadSingleton, -javaagent:bin/tools/ws-javaagent.jar, -jar, bin/tools/ws-server.jar, dataset-service] with console output: Launching dataset-service (WebSphere Application Server 16.0.0.4/wlp-1.0.15.cl160420161113-0206) on IBM J9 VM, version pwa6480sr3-20160428_01 (SR3) (en_US)  [AUDIT   ] CWWKE0001I: The server dataset-service has been launched.  [ERROR   ] CWWKF0003E: An exception was generated when installing or uninstalling bundle INSTALL reference:file:/C:/Program%20Files/ibm/cognos11_0_6/analytics/wlp/usr/extension/lib/com.ibm.bi.logging.jar (resolved from: com.ibm.bi.logging.api/[1.0.0,2.0.0)@file:/C:/Program%20Files/ibm/cognos11_0_6/analytics/wlp/usr/extension/lib/com.ibm.bi.logging.jar). Exception: java.lang.NullPointerException     at com.ibm.ws.kernel.feature.internal.Provisioner$1.handle(Provisioner.java:259)     at [internal classes]   

I highlighted the out of memory error above because that sticks out to me that it's failing due to insufficient resources, it just seems odd that it's happening all of a sudden.  I've tried reducing the heap from 4096 to 2048 but that hasn't allowed me to get any further along.  When starting it up it takes a good 10 minutes.

I've also tried changing the port to 9350 to see if it was something to do with that with no luck either.

Any insight would be most appreciated!

[MFGF]

Hey Folks,
We've begun our 11.0.6 upgrade investigation and have run into an odd issue.  We first installed and configured everything without the optional gateway, got all the audit packages and samples up and running and everything was working great.  We then configured IIS and even that was working really well.  Our instance was cycled after that and now when starting the service the Query Service fails to start.  Everything else comes up successfully and we can navigate through the portal without issue (via SG or the IIS gw) but when we go to run a report or test a data source it fails with the following error;

DPR-ERR-2002 Unable to execute the request because there were no connections to the process available within the configured time limit.

Taking a look through the cogserver reveals the following information;

QueryService failed to start with the following command-line arguments: [C:\Program Files\ibm\cognos11_0_6\analytics\jre\bin\java.exe, -Xms1024m, -Xmx8192m, -Xmnx4096m, -Xmns512m, -server, -Xjit:disableLoopVersioner, -Xscmx100m, -Xshareclasses:cachedir=../javasharedresources,name=cognos10%u,nonfatal, -Xmso512K, -Xcompressedrefs, -verbose:gc, -Xverbosegclog:C:\Program Files\ibm\cognos11_0_6\analytics\logs\XQE\dq_verbosegc_%Y%m%d.%H%M%S.%pid.log,1,1000, -Xgcpolicy:gencon, -Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec=cmd.exe /c del /f /s \"C:\Program Files\ibm\cognos11_0_6\analytics\data\.xqe.%pid\" > nul, -DCOG_ROOT=C:/Program Files/ibm/cognos11_0_6/analytics, -DParentProcessID=8448, -Dcubingservices_home=C:/Program Files/ibm/cognos11_0_6/analytics/v5dataserver, -Dcubingservices_data=C:\Program Files\ibm\cognos11_0_6\analytics\data, -DParentProcessPort=49365, -Djava.library.path=C:/Program Files/ibm/cognos11_0_6/analytics/bin64;, -DenableTraceServer, -ea, -Dorg.dom4j.QName.singleton.strategy=org.dom4j.util.PerThreadSingleton, -Dorg.dom4j.DocumentFactory.singleton.strategy=org.dom4j.util.PerThreadSingleton, -javaagent:bin/tools/ws-javaagent.jar, -jar, bin/tools/ws-server.jar, dataset-service] with console output: Launching dataset-service (WebSphere Application Server 16.0.0.4/wlp-1.0.15.cl160420161113-0206) on IBM J9 VM, version pwa6480sr3-20160428_01 (SR3) (en_US)  [AUDIT   ] CWWKE0001I: The server dataset-service has been launched.  [ERROR   ] CWWKF0003E: An exception was generated when installing or uninstalling bundle INSTALL reference:file:/C:/Program%20Files/ibm/cognos11_0_6/analytics/wlp/usr/extension/lib/com.ibm.bi.logging.jar (resolved from: com.ibm.bi.logging.api/[1.0.0,2.0.0)@file:/C:/Program%20Files/ibm/cognos11_0_6/analytics/wlp/usr/extension/lib/com.ibm.bi.logging.jar). Exception: java.lang.NullPointerException     at com.ibm.ws.kernel.feature.internal.Provisioner$1.handle(Provisioner.java:259)     at [internal classes]   

I highlighted the out of memory error above because that sticks out to me that it's failing due to insufficient resources, it just seems odd that it's happening all of a sudden.  I've tried reducing the heap from 4096 to 2048 but that hasn't allowed me to get any further along.  When starting it up it takes a good 10 minutes.

I've also tried changing the port to 9350 to see if it was something to do with that with no luck either.

Any insight would be most appreciated!

Hi,

There is a separate memory pool for the query service - you can find this in the properties of the Query Service in the Admin Console - you can set an initial JVM heap size and a heap size limit. My guess is that the limit is too low? This is entirely separate from the jvm heap size for the main CA servlet container (the one you reference above).

How much memory does your server have? How much is being taken by the java processes when Cognos is started?

MF.

[johnck77]

Hi gohabsgo

I have exactly the same issue, trying to start Cognos from Cognos Configuration and it fails at 'QueryService', 'StartService', 'Sucess' you can log on to the server and so all sorts of thing but as soon as you try to run a report errors all over the place

The annoying thing is sometimes Cognos fails to start but sometimes Cognos will start with out any problem. Even more confusing is that I have two identical servers and on server 1 most of the time Cognos starts ok, probably 80% of the time and the other 20% it stops at QueryService, server 2 is the other way round 90% of the time it fails at QueryService, but it i'm lucky it will start


MFGF is usually correct so I'm going to look in to what they say and will report back what happens

[johnck77]

I know this is a quick response to my own post only moments ago, but I was running things in the back ground and I promised to reply if I found anything. OK so this is what I have

Trying to start Cognos from Cognos Configuration and it fails at 'QueryService', 'StartService', 'Sucess' eventually it comes back with CFG_ERR_0106 IBM Cognos Configuration did not receive a response from the IBM Cognos Service  in the time allotted. Check the IBM Cognos service is available and properly configured.

The error is a generic Cognos service did not start error and searching for it the error lead me to this page http://www-01.ibm.com/support/docview.wss?uid=swg21646350   and this page http://www-01.ibm.com/support/docview.wss?uid=swg21965068 neither are specifically for 11 but they are a good place to start.

For me I think I have managed to resolve it using the pointers from the pages above, but only time will tell. What I have done is

On server 1 which worked for 80% of restarts, in Cognos Configuration removing the domain part of the server names seems to change worked. ie changing from server.domain to just server. Please don't ask me why, I know that Information Server is a bit picky about this so I thought I'd try it and it appears to be working.

On server 2, identical hardware and software to server 1, which only started properly it I was lucky and trees moved in a certain way. In Cognos Configuration I changed the "Maximum memory for WebSphere Liberty Profile in MB" from 4096 to 5120  and it appears to be working.

MFGF - I haven't tried changing things in the admin console as I was already playing with this when I typed my first message

[MFGF]

Thanks for the update! This might prove really valuable for others in the same situation :)

MF.