[C8] Log Service Failed

[jguevin]

Running C8 for about 3 months now and on occasion (couple times every other week), get the following error message:

9301   4992   2006-11-29 09:37:29.272   -5 Thread-2   LOGSV   603   1   server.Audit.IPF   StartService   LogService      Failed   Log Server socket error   <parameters><item name="Port"><![CDATA[11002]]></item><item name="Error"><![CDATA[Address already in use: Cannot bind]]></item></parameters>

This error shuts the log service down and no longer captures the things I need to keep track of users.  C8 still seems to work fine for our authors, consumers etc...  A re-start of the Cognos 8 service is required in order to get the log back.  I understand that the port is being used by something else, but can't figure it out.  I have run netstat on the server and this shows nothing using that port.  I have talked with our IT staff and they don't run anything that causes this.  I have asked cognos support and they had no answers.

Has anyone else had this issue?  My therory is that somewhere out in my user base someone is leaving a session on Cognos8 open and inactive until it times out ( I think 30 minutes).  Then they try to do something in Cognos8 and it causes the log service to fail because the port is being "tied" up by their inactive session.  Anyone have thoughts on this?

Thanks
Jeff

[Blue]

Have you tried changing the assigned port number in configuration?

[jguevin]

yes, more than once .  The issue has happened with different port numbers.

Jeff

[lindero]

have must be sure before you start the service that there are no BIBUS processes still exists.

I had the same problem. The processes bind the port for logging.

[sir_jeroen]

is file logging the only type of logging you do? Or do you also e.g. use windows event logging?

[jguevin]

only the file logging.  no other logging is going on.  I mentioned earlier that I think it happens when someone leaves their session of cognos 8 open and it times out...this has caused it more than once and I purposely have left my session open to see and indeed it shuts the log service down.

[tdyoda]

I've seen this exact error occur when BIBus is serving core dumps.  Are you generating core?

[jguevin]

I'm not sure what you mean by "generating core", could you please explain?

thanks
Jeff

[tdyoda]

BI Bus will generate structured dumps in the /bin directory by default.  They start with "BI" and have the file extension of "dmp"

Look in that directory and see if you see any recent .dmp files.

[jguevin]

thanks for the clarification.  We did have these .dmp files created in the bin directory, but it was the result of one of our RS reports that contained 30+ queries and when we added one more query and ran the report the .dmp file was created and C8 crashed.  We had a different case open with Cognos on this one and they showed me where to turn off the .dmp from being generated.  It was like we reached the limit of what RS could do by adding this one more query.

The difference with this is that when the log service fails we can still use C8 as if nothing happened.  I would only lose the info I collect from the log files because they were "shut off".

thanks
Jeff

[hendrixski]

Don't core dumps happen AFTER a system crashes, as a way of capturing the state of the program when it crashed?  It's an error message in a file.  How is stopping an error message from being generated supposed to stop the error from happening? 

A good followup question is: are these .dmp files are human readable, or if not (I assume they're not) does Cognos has a tool with which to decipher them, is it perhaps the same logging tool you're using?  If Cognos doesn't, does one of its partners have such a tool, or is there an open source tool that does?  This would give you more information on what specifically is causing it to crash.

[tdyoda]

I was told by Cognos support that these structured exception dumps are the state of everything in their code at the time of the failure, just like any other C++ structured dump.  I was also told that they have tools that they use to make them readable.  I have not been able to read them using a myriad of tools, hence my assumption that they were core dumps instead of a structured exception dump.

This approach is their way of leaving a bread crumb trail to find the root cause, and a standard coding practice for any professional developer.

We were receiving dmp files due to some environment variables not being quite right.  Orphaned BI Bus processes were caused by the exceptions - they locked the sockets that the logging process was using which blocked logging - exactly as was shown in the error message shown below:

9301   4992   2006-11-29 09:37:29.272   -5 Thread-2   LOGSV   603   1   server.Audit.IPF   
StartService   LogService      Failed   Log Server socket error   <parameters><item name="Port"><![CDATA[11002]]></item><item name="Error"><![CDATA[Address already in use: Cannot bind]]></item></parameters>

Just change the date/time and I'm looking at the same error message.

We did a reboot for good measure to start, but now we find that stopping the service and checking for orphaned BIBus processes (ie. after the service is stopped, there should not be any BIBUS processes; if there are they are orphans) is all that is necessary.  Restarting the service clears it up.

What is your memory model and java heap size?

[cognosjon]

We had this exact same problem with the logging service and I contacted Cognos Support. After some investigation the following solution fixed our problem.

Solution:

Change  "Enable TCP?" to True in C8  configuration

Steps:

1. Launch C8 configuration

2. Click logging, set "Enable TCP?" to True

3. Restart Cognos 8 service


Hope it helps


Jonathan ;D

[jguevin]

Thanks Jonathon,
I enabled the TCP to 'true' so we shall see if the issue happens again.
regards
Jeff

[jguevin]

just about 2 weeks now and no log service failure.  thanks again

[dmccoy]

Setting TCP to true fixed my problem with logging shutting off. Thanks.