DQM fails and throws errors

[navissar]

Boy, do I need some help.  :)
I have two Cognos 10.2.1 environments (Namely, QA and Prod). They are, as far as I can tell, identical in all aspects.
On QA everything works super. On Prod, however, I'm having issues with DQM. The query service starts, and then throws errors t the log until it maxes out on memory and dies horribly, leaving a nice dump file behind.
This is the error that repeats in the cogserver log:

Code Select Expand

192.168.10.104:9300 4720 2014-07-01 14:14:54.802 +2 62 XQE 0 4 Audit.RTUsage.qs.XQE.QueryService Response Query Service returned response for "configure" request from "unknown" report.
192.168.10.104:9300 7080 2014-07-01 14:15:55.902 +2 na na Timer-50 DISP 5000 1 Audit.Other.dispatcher.DISP.pogo pogo com.cognos.pogo.reportservice.ProcessFacade Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message></messages> QueryService process appears to have stopped. Signalling Handler...
192.168.10.104:9300 7080 2014-07-01 14:16:04.121 +2 na na com.cognos.pogo.async.service.connection.bibustkserver.BIBusTKServerConnectionFactory-queryProcessManager-ProcessReaper DISP 5000 1 Audit.Other.dispatcher.DISP.pogo pogo com.cognos.pogo.reportservice.ProcessFacade Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Failed to send HTTP request or read HTTP response</messageString></message></messages> Query Service - V5DataServerProcessFacade.sendStop failure. Failed to send HTTP request or read HTTP response java.io.IOException: Failed to send HTTP request or read HTTP response at com.cognos.pogo.http.ServerConnection.executeMethod(ServerConnection.java:796) at com.cognos.pogo.http.ServerConnection.sendRequest(ServerConnection.java:250) at com.cognos.xqe.cubingservices.CubingServicesComponentFactory$V5DataServerProcessFacade.sendStop(CubingServicesComponentFactory.java:340) at com.cognos.pogo.reportservice.ProcessReaper.destroyProcesses(ProcessReaper.java:199) at com.cognos.pogo.reportservice.ProcessReaper.safeRun(ProcessReaper.java:260) at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61) Caused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:80) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:369) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:176) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402) at java.net.Socket.connect(Socket.java:590) at java.net.Socket.connect(Socket.java:539) at java.net.Socket.<init>(Socket.java:436) at java.net.Socket.<init>(Socket.java:291) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at com.cognos.pogo.http.ServerConnection.executeMethodSynchronously(ServerConnection.java:827) at com.cognos.pogo.http.ServerConnection.executeMethod(ServerConnection.java:777) ... 5 more

This is the beginning of the error from the XQE log:

Code Select Expand


java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: ynvcog01; nested exception is: java.io.IOException: Reason: java.net.ConnectException: Connection refused: connect



I'm close to tears by now. I have looked everywhere I can, I searched these error messages, I've done some things I'm not proud of (Such as reinstalling the environment). Has anyone ever encountered anything like this?

[navissar]

Well, I went for the horrible but obvious solution of copying the test environment to the production (God bless VM).
Sometimes solving the issue is more important than understanding it...

[MFGF]

Well, I went for the horrible but obvious solution of copying the test environment to the production (God bless VM).
Sometimes solving the issue is more important than understanding it...

Sorry - late to the party! My only excuse is I was on holiday...

I'm wondering if you had enough memory allocated on the Prod box? It might have been a useful exercise to copy the cogstartup.xml file from QA to Prod, then re-configure with the Prod hostname. Too late now though...

Glad you're all sorted!

MF.

[Ammus1234]

Hi Nimrod,

Have you found any reason for this issue. I am facing same problem with DQM.
Found these errors from logs. My version is C10.2.1 FP2

nov. 20, 2014 6:36:57 AM ClientCommunicatorAdmin restart
AVERTISSEMENT: Failed to restart: java.io.IOException: Failed to get a RMI stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: <server>; nested exception is:
   java.io.IOException:  Raison : java.net.ConnectException: Connection refused: connect]
nov. 20, 2014 6:36:58 AM RMIConnector RMIClientCommunicatorAdmin-doStop
AVERTISSEMENT: Failed to call the method close():java.rmi.ConnectIOException: Exception creating connection to: 10.2.11.116; nested exception is:
   java.io.IOException:  Raison : java.net.ConnectException: Connection refused: connect
nov. 20, 2014 6:36:58 AM ClientCommunicatorAdmin Checker-run
AVERTISSEMENT: Failed to check connection: java.net.SocketException: Connection reset
nov. 20, 2014 6:36:58 AM ClientCommunicatorAdmin Checker-run
AVERTISSEMENT: stopping

2014-11-20 06:37:04.709 ERROR [om.cognos.pogo.reportservice.ProcessFacade] com.cognos.pogo.async.service.connection.bibustkserver.BIBusTKServerConnectionFactory-queryProcessManager-ProcessReaper: Query Service - V5DataServerProcessFacade.sendStop failure.  java.io.IOException: Failed to send HTTP request or read HTTP response
   at com.cognos.pogo.http.ServerConnection.executeMethod(ServerConnection.java:797)
   at com.cognos.pogo.http.ServerConnection.sendRequest(ServerConnection.java:251)
   at com.cognos.xqe.cubingservices.CubingServicesComponentFactory$V5DataServerProcessFacade.sendStop(CubingServicesComponentFactory.java:325)
   at com.cognos.pogo.reportservice.ProcessReaper.destroyProcesses(ProcessReaper.java:183)
   at com.cognos.pogo.reportservice.ProcessReaper.safeRun(ProcessReaper.java:237)
   at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)
Caused by: java.net.ConnectException: Connection refused: connect
   at java.net.DualStackPlainSocketImpl.connect0(Native Method)
   at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
   at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:358)
   at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:219)
   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:201)
   at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:180)
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   at java.net.Socket.connect(Socket.java:616)
   at java.net.Socket.connect(Socket.java:565)
   at java.net.Socket.<init>(Socket.java:432)
   at java.net.Socket.<init>(Socket.java:287)
   at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
   at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at com.cognos.pogo.http.ServerConnection.executeMethodSynchronously(ServerConnection.java:826)
   at com.cognos.pogo.http.ServerConnection.executeMethod(ServerConnection.java:778)
   ... 5 more

[cognostechie]

It says Exception creating connection to: 10.2.11.116  . Which server is that? Can you ping it from all servers?

[Ammus1234]

I guess it is Ok with FP4.