Cognos services in state of starting - hangs the environment recycle process

[jeffowentn]

So, we have 12 servers on a mix of Win 2003 and Win 2008 - some virtual and some physical (all located in the same building).

We are running Cognos 8.4.1 FP3 for BI and FP2 for Planning. 

We currently have a couple of jobs scheduled to run as BI Macros every hour, 168 hours of the week.  We also have our environment scheduled to recycle and reboot every Mon., Wed., and Fri., at 3a.  We are seeing occassions when the batch script used to stop all services, reboot the servers, and start all services get stuck in a state of "starting," regardless of whether the script is attempting to stop all service or attempting to start all services.  This causes a problem b/c the script cannot move forward.  It is coded to kill the process and then attempt to either stop or start, again, depending on what phase it is in for the script.

Here's what I believe I learned in this morning's production outage:  something is causing the service to automatically start back up even though I've killed the services and even though the services are set to manual start (not automatic start).

This the only thing I can think of:  is it possible that one of the jobs is kicking off just before the recycle and reboot script starts such that a Cognos job is attempting to be processed in the midst of shutting down the environment?  Could this job be causing services to automatically start?  If so, one quick fix is to simply push the script execution earlier or later to either make sure it starts before the job gets kicked off or make sure the job is finished before the script gets kicked off.  I don't have a problem doing either, but I was hoping someone might have some insight as to whether this will resolve the issue.  This continues to be a problem for us, and we have dealt with it for several months, in production.

Any help is greatly appreciated.  Let me know if you need any further info.

Jeff