All reports stop working with error message RSV-SRV-0040 until service restart

[Gollapudi]

Hi,

A strange thing has been happening without Cognos Analytics, all of a sudden we get a error like RSV-SRV-0040 for a report and all the repots stops working and give us the same error until we start the service, no specific report is triggering this issue as we did a stress test.

Just wondering if anyone has got the similar issue and had any crack at it so far.

RSV-SRV-0040   The report server encountered an internal error. Check additional information associated with this error message. If cause of problem cannot be ascertained, increase the logging level in the IBM Cognos administration tool and reproduce the conditions that caused the error. If the problem persists, see the problem determination information on the IBM Cognos Support Portal page at http://www-947.ibm.com/support/entry/portal/Overview/Software/Information_Management/Cognos_Business_Intelligence.   DetailsCCLThreadResourceError:-1:Fatal: RSV-SRV-0042 Trace back:RSReportService.cpp(774): CCLThreadResourceError: CCL_CAUGHT: RSReportService::processImpl()RSReportServiceMethod.cpp(254): CCLThreadResourceError: CCL_RETHROW: RSReportServiceMethod::process(): asynchRun_RequestRSASyncExecutionThread.cpp(887): CCLThreadResourceError: RSASyncExecutionThread::checkExceptionRSASyncExecutionThread.cpp(331): CCLThreadResourceError: CCL_CAUGHT: RSASyncExecutionThread::runImpl(): asynchRun_RequestRSASyncExecutionThread.cpp(932): CCLThreadResourceError: CCL_RETHROW: RSASyncExecutionThread::processCommand(): asynchRun_RequestExecution/RSRenderExecution.cpp(585): CCLThreadResourceError: CCL_RETHROW: RSRenderExecution::executeAssembly/RSDocAssemblyDispatch.cpp(339): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSLayoutAssembly.cpp(79): CCLThreadResourceError: CCL_RETHROW: RSLayoutAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSReportPagesAssembly.cpp(185): CCLThreadResourceError: CCL_RETHROW: RSReportPagesAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(384): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSPageAssembly.cpp(314): CCLThreadResourceError: CCL_RETHROW: RSPageAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(384): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSTableRowAssembly.cpp(177): CCLThreadResourceError: CCL_RETHROW: RSTableRowAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(384): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSTableCellAssembly.cpp(151): CCLThreadResourceError: CCL_RETHROW: RSTableCellAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSTableAssembly.cpp(119): CCLThreadResourceError: CCL_RETHROW: RSTableAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSTableRowAssembly.cpp(177): CCLThreadResourceError: CCL_RETHROW: RSTableRowAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(384): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSTableCellAssembly.cpp(151): CCLThreadResourceError: CCL_RETHROW: RSTableCellAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSTableAssembly.cpp(119): CCLThreadResourceError: CCL_RETHROW: RSTableAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSTableRowAssembly.cpp(177): CCLThreadResourceError: CCL_RETHROW: RSTableRowAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(384): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSTableCellAssembly.cpp(151): CCLThreadResourceError: CCL_RETHROW: RSTableCellAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSTableAssembly.cpp(119): CCLThreadResourceError: CCL_RETHROW: RSTableAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSTableRowAssembly.cpp(177): CCLThreadResourceError: CCL_RETHROW: RSTableRowAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(384): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchAssemblyAssembly/RSTableCellAssembly.cpp(151): CCLThreadResourceError: CCL_RETHROW: RSTableCellAssembly::assembleAssembly/RSDocAssemblyDispatch.cpp(448): CCLThreadResourceError: CCL_RETHROW: RSDocAssemblyDispatch::dispatchChildrenAssemblyForwardAssembly/RSSingletonAssembly.cpp(126): CCLThreadResourceError: CCL_RETHROW: RSSingletonAssembly::assembleAssembly/RSAssembly.cpp(699): CCLThreadResourceError: CCL_RETHROW: RSAssembly::createListIteratorRSResultSetIterator.cpp(1513): CCLThreadResourceError: CCL_RETHROW: RSResultSetIterator::retrieveDataQFSPartialDataset.cpp(80): CCLThreadResourceError: CCL_RETHROW: QFSPartialDataset::GetEdgeIteratorSource/QEDmSqlResultSet.cpp(1942): CCLThreadResourceError: CCL_RETHROW: QESource/QEDMSqlCursorScroll.cpp(449): CCLThreadResourceError: CCL_RETHROW: Source/AQE_UDAFacade.cpp(980): CCLThreadResourceError: CCL_RETHROW: AQE_UDAFacade::sqlOpenResult CCLErrorapicursr.cpp(552): CCLThreadResourceError: CCL_RETHROW: sqlOpenResult CCLErrorapigbl.cpp(3927): CCLThreadResourceError: APICursorOpenThread::joinsgiWrapper.cpp(989): CCLThreadResourceError: CCL_RETHROW: sgiAOpen CCLThrowablesgiCursor.cpp(275): CCLThreadResourceError: CCL_RETHROW: SgiCursor::asyncOpen CCLThrowablecommon/sgiOdbcCursor.cpp(330): CCLThreadResourceError: CCL_RETHROW: SgiOdbcCursor::doOpen CCLThrowablecommon/sgiOdbcCursor.cpp(1050): CCLThreadResourceError: CCL_RETHROW: SgiOdbcCursor::openForNonMetaRqst CCLThrowableThreads/CCLThread.cpp(176): CCLThreadResourceError: CCL_THROW: CogThread interface or unknown error

Apart from this, when we restart Cognos service I an see the following log in Event Viewer Windows logs>> Application

Faulting application name: cogbootstrapservice.exe, version: 11.1.5.2, time stamp: 0x5daf2515
Faulting module name: ntdll.dll, version: 10.0.14393.4350, time stamp: 0x606eaf8d
Exception code: 0xc0000374
Fault offset: 0x00000000000f70d3
Faulting process id: 0xdcc
Faulting application start time: 0x01d7419e5da96a6e
Faulting application path: C:\Program Files\ibm\cognos\analytics\bin64\cogbootstrapservice.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 015eaf12-f161-4475-9cdf-a3c6d5a8ff2a
Faulting package full name:
Faulting package-relative application ID:

[MFGF]

Hi,

A strange thing has been happening without Cognos Analytics, all of a sudden we get a error like RSV-SRV-0040 for a report and all the repots stops working and give us the same error until we start the service, no specific report is triggering this issue as we did a stress test.

Just wondering if anyone has got the similar issue and had any crack at it so far.

Apart from this, when we restart Cognos service I an see the following log in Event Viewer Windows logs>> Application

Hi,

This is a very generic error message and it doesn't really tell us a lot other than that the server had a serious problem. What OS are you running Cognos Analytics on? How much memory does the server have? Is it possible the server is out of memory?

The second issue looks like you might have a corrupted dll on your system. Have you tried running a file check? Worst case you make need to reinstall that dll.

Cheers!

MF.

[Gollapudi]

Hi MFGF,

Thanks for the response.

OS: Windows Server 2016 DataCenter
Memoery: 264GB RAM and never ran out of memory and highest memory consumption so far was never over 60%
8 vCPU @ 2.4GHz

Concurrent users are about 12 including 3 power users and 65 active Cognos users overall.

About file check, can you tell me if there is something specific for Cognos Analytics.
I am thinking of installing Fix Packs (11.1.6-IF001 and 11.1.6-IF002) hopefully that should fix the corrupted dll files if thats the case

Let me know if you can think of any other possibility.

[MFGF]

Hi MFGF,

Thanks for the response.

OS: Windows Server 2016 DataCenter
Memoery: 264GB RAM and never ran out of memory and highest memory consumption so far was never over 60%
8 vCPU @ 2.4GHz

Concurrent users are about 12 including 3 power users and 65 active Cognos users overall.

About file check, can you tell me if there is something specific for Cognos Analytics.
I am thinking of installing Fix Packs (11.1.6-IF001 and 11.1.6-IF002) hopefully that should fix the corrupted dll files if thats the case

Let me know if you can think of any other possibility.

It sounds like your server is more than up to the task, so not likely to be a memory issue here. Given the generic nature of the error it's definitely worth you opening a case with IBM support I'd suggest.

Re the file check - no, there's nothing specific for CA - just a regular windows file check of the filesystem. The (possibly) corrupted dll looks like a Windows dll not a CA dll, so I doubt installing a CA interim fix will change anything on that score.

Cheers!

MF.