Major issue with Java / Cognos partial processes on Win 2003 server

[blom0344]

For a couple of days now we are trying to fix a Cognos 8.1 issue with one of our clients.

Their first call hinted at the Cognos services not being up in the air. However both C8 configuration as well as Windows services indicated Cognos was running. A restart from C8 configuration ran into a time-out (with obviously no partial processes and detail rendered)

So far we have this:

1. The cbs_run log shows:
PingChildProcess ping loop: process "catalina" is not active, so restarting it. (again and again)
2. We dabbled around with the JAVA_HOME settings as the KB suggests, but to no avail.
3. Using cogconfig.bat - s from a cmd box shows that the JAVA_HOME is correctly found, but this still runs into a time-out.
4. Installed an additional Cognos instance on the server using different ports (as we have with other clients) , but this yields the same errors.
5. Installed a newer Java SDK and pointed the JAVA_HOME to it, but it looks as though this is not compatible with C8.1

We have seen every hint from the KB, but to no avail.

Anyone suggestions? [We have thefeeling that it may have to do with the Windows update , cause the problem arose after the server reboot after the latest Win 2003 updates]


 

[smiley]

When you open a cmd and type 'java -version" does it match the 1.4.2 from your C8.1?

[kolonell]

that seems odd .. Have checked the file permissions for the Cognos 8 install folder ? the Service Account  might not have sufficient permissions to launch The Java process. Other than that you might want to try with DEP disabled.

[blom0344]

When you open a cmd and type 'java -version" does it match the 1.4.2 from your C8.1?

returned is: 1.4.2_09
Java is a little out of my competence. Any idea if this is the/a 8.1 compatible one?

[blom0344]

that seems odd .. Have checked the file permissions for the Cognos 8 install folder ? the Service Account  might not have sufficient permissions to launch The Java process. Other than that you might want to try with DEP disabled.

I am not sure about DEP. From the help info it only works on Win2003 server SP1 and by default it should only " By default, DEP is only turned on for essential Windows operating system programs and services" On our develop machine it is set for all programs and Cognos / Java is no problem at all.
Useful tip nonetheless!!

[ducthcogtechie]

If you cannot see a java.exe proces when you start C8 (see windows taskmanager), then that is your problem.
To help this the hard way, add the following entry to your server environment path at the BEGINNING of all paths:

C:\Program Files\Cognos\c8\bin\jre\1.4.2\bin

A server reboot could be required to pic this up.

[blom0344]

Do you mean that the java process has to be running prior to starting Cognos?

edit:

When trying to start Cognos with cogconfig.bat -s the java.exe appears, but in task manager it can be seen loading till about 32Mb. Then it stops from loading into an executable that should take up some hundreds of Mb and it disappears from task manager when Cognos runs into the time-out

[ducthcogtechie]

no, starting the C8 service will spawn a java.exe.
If you do not see a java.exe loaded via windows task manager, it means C8 cannot find it's way to the java.exe (the one that comes with C8)

[ducthcogtechie]

I think your 32mb is a java.exe from the wrong location.
Look at my previous post on hardcoding the path to the correct java.exe 

[kolonell]

> no, starting the C8 service will spawn a java.exe.

That is true even with the silent start option. cogconfig (Cognos Configuration) is a java program (I think) so that is why you see the
32Mb java.exe . Can you successfully start the service using the startup.bat (c8/bin folder) ? if so that would indicate something is wrong with the service account (startup.bat will start C8 as the account currently logged on with ). it will also give you more info (OS related) on possible problems during startup.

Blom, Did you check the permissions on the C8 install folder (more specifically for the SYSTEM user account)? I think you did but just for my peace of mind ;-)

[smiley]

To rule out where the 32mb guy is coming from, open a cmd and type:

wmic process list >c:\list.txt

Open with notepad, with the option "Word Wrap" ticked off.

[blom0344]

> no, starting the C8 service will spawn a java.exe.

That is true even with the silent start option. cogconfig (Cognos Configuration) is a java program (I think) so that is why you see the
32Mb java.exe . Can you successfully start the service using the startup.bat (c8/bin folder) ? if so that would indicate something is wrong with the service account (startup.bat will start C8 as the account currently logged on with ). it will also give you more info (OS related) on possible problems during startup.

Blom, Did you check the permissions on the C8 install folder (more specifically for the SYSTEM user account)? I think you did but just for my peace of mind ;-)

We tried to use startup.bat , but with no success.

The cbs_start.log contains:

Mon May 04 10:17:56 2009  INFO  t[1234] CBSNTService Attempt to starting service "Cognos 8"
Mon May 04 10:17:56 2009  INFO  t[1234] CBSNTService Starting service "Cognos 8"
Mon May 04 10:17:56 2009  INFO  t[1234] CBSNTService Waiting for service to start...
Mon May 04 10:23:01 2009  INFO  t[1234] CBSNTService Started service "Cognos 8"

Mon May 04 10:23:02 2009  INFO  t[1234] CBSBootstrapService attempt to load config from "../bin\./bootstrap_win32.xml"
Mon May 04 10:23:02 2009  ERROR t[1234] CBSSocketCommand failed to connect, CAM error: <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 An error was returned from the underlying socket implementation.</errorMessage><errorStack><errorCode>10061</errorCode><errorMessage>Could not connect the socket, errno: 0x274d(10061)</errorMessage></errorStack></errorDetail>
Mon May 04 10:23:05 2009  ERROR t[1234] CBSSocketCommand failed to connect, CAM error: <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 An error was returned from the underlying socket implementation.</errorMessage><errorStack><errorCode>10061</errorCode><errorMessage>Could not connect the socket, errno: 0x274d(10061)</errorMessage></errorStack></errorDetail>
Mon May 04 10:23:07 2009  ERROR t[1234] CBSSocketCommand failed to connect, CAM error: <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 An error was returned from the underlying socket implementation.</errorMessage><errorStack><errorCode>10061</errorCode><errorMessage>Could not connect the socket, errno: 0x274d(10061)</errorMessage></errorStack></errorDetail>
Mon May 04 10:23:08 2009  INFO  t[1234] CBSBootstrapService testReadyByPingingWin - child processes are not ready, stop trying.

[kolonell]

Hi Blom

The only other thing I can think of is that there isn't enough Memory available to start up the Java.exe (as in it cannot allocate the memory in one continuous piece ). Maybe try reducing the "Max dispatcher Memory in Mb" setting for the service and try starting it again.

Or if all comes to worse, recycle the Server just in cause memory has become too fragmented .

[blom0344]

We have always worked with the standard 768 for maximum memory size. The Server is a 4 GB machine.
do you mean we should lower this to 704 - 640 - 576 etc stepwise as suggested by the KB:

KB 1016472

Steps:
Open Cognos Configuration
Check the value for Maximum Memory in MB under Cognos 8 Service/<your_instance_name>
Change the value to 1024 MB or less if needed (try 64 MB steps)
If steps 1 to 3 don't help, try starting the services by running the "startup.bat" file that is in the c8_location/bin directory.

[We actually tried setting it to 1024 MB , cause we were unsure of its function]

[kolonell]

We actually tried setting it to 1024 MB , cause we were unsure of its function

The "Dispatcher Memory in Mb" is used to specify the max amount of memory the Java Heap can occupy ( http://en.wikipedia.org/wiki/Java_Virtual_Machine_heap ). If this is set too high
and/or can't be allocated in one continuous piece of memory the JVM will not launch (due to the Garbage Collector).

What is the memory use on the server atm ? Typically 1024MB isn't a problem on Win32 platforms but I wouldn't go much higher than that (due to 32-bit limitation) . I'd try with 512MB first

[blom0344]

I changed max amount of memory from 678 to 512.

Checking settings with CTRL+F3:

I notice that 2 system entries are no longer present:

java.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol
noticecast.server_dir=C:\Program Files\Cognos\c8\sds\conf

These entries do not appear in our own C8 configurations.

Start Cognos from configuration, the services really do start normally now.
I pinch myself .. Stop cognos services
Set max memory to 768 again , the services again do not start normally (timeout)

Check the system entries:

java.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol
noticecast.server_dir=C:\Program Files\Cognos\c8\sds\conf

are back again.

Set memory to 512 and save , system entries are still there.
close and reopen configuration system entries are gone..

Start Cognos and all runs well again

Slap myself , pour another coffee    :o

[kolonell]

Slap myself , pour another coffee    :o

Hold the slap ;-)  Redbull does it better for me

Next thing would be to check what is using up the memory on that 4Gig RAM machine. You should be able to launch C8 on 768MB 

[blom0344]

I appreciate your input for resolving this issue! Though I am still baffled why reducing the memory setting has worked. But hey, this is cognos after all  ;D

[by the way, the server is consuming about 900Mb with java.exe consuming about 260Mb on a 4 Gig machine]