Java error in Scheduler when running scheduled Export jobs

[jriley]

Hello,

We're running Cognos 8.3 in a multi-server environment, APP server and 2 dispatchers.  Oracle 10g backend.  We have Fix Pack 3 on our App Server and one of our dispatchers.  We have Fix Pack 4 on the other dispatcher.
Java version 1.0.6_26 on App Server. 

We have intermittent problems with these scheduled exports running successfully.  Other scheduled jobs, i.e. Reports, run successfully without this problem. 

We were told that the issue is if the export job is sent from App Server (FP3) to Dispatcher (FP4), the job won't run.  We have turned on logging and have tracked these issues for our 2 nightly export jobs.  One for Report files export, the other for CS and User export .  Here are the entries in Cogserver.log file for these two jobs that fail repeatedly.  I have watched the server activities at the times these jobs fail and there is no other activity which affected these jobs and would have kicked them over the the 2nd dispatcher that I could see.  So not sure why this happens.  These jobs ran all last week per schedule but have not run for the last 2 days with these errors. 

We've restarted services, re-created jobs, re-scheduled, eliminated possible conflicts.  The only thing we haven't done is update the App server and first dispatcher to fix pack 4.  (This was all done before my time.) 

Any thoughts?  Any help would be greatly appreciated. 

Thanks,
Joe

192.168.xxx.xxx:9300      340       2011-07-26 21:23:10.016            -5                                                         EMS.Scheduler            EMS     720       2          Audit.RTUsage.ems.EMS                                                           Scheduler - error calling runTask: nulljava.lang.NullPointerException
            at noticecast.runtime.RunTimeManager.storeTaskScheduleFromThread(RunTimeManager.java:861)
            at noticecast.runtime.RunTimeManager.runTask(RunTimeManager.java:627)
            at noticecast.runtime.RunTimeManager.runTask(RunTimeManager.java:596)
            at noticecast.scheduling.nc.TaskScheduleImpl.runTask(TaskScheduleImpl.java:391)
            at noticecast.scheduling.nc.TaskScheduleImpl.runTask(TaskScheduleImpl.java:395)
            at noticecast.scheduling.nc.NCSchedulable.runTask(NCSchedulable.java:67)
            at noticecast.scheduling.core.Scheduler.runTasks(Scheduler.java:362)
            at noticecast.scheduling.core.Scheduler.run(Scheduler.java:274)
            at java.lang.Thread.run(Unknown Source)


192.168.xxx.xxx:9300      340       2011-07-26 22:12:10.085            -5                                                         EMS.Scheduler            EMS     720       2          Audit.RTUsage.ems.EMS                                                           Scheduler - error calling runTask: nulljava.lang.NullPointerException
            at noticecast.runtime.RunTimeManager.storeTaskScheduleFromThread(RunTimeManager.java:861)
            at noticecast.runtime.RunTimeManager.runTask(RunTimeManager.java:627)
            at noticecast.runtime.RunTimeManager.runTask(RunTimeManager.java:596)
            at noticecast.scheduling.nc.TaskScheduleImpl.runTask(TaskScheduleImpl.java:391)
            at noticecast.scheduling.nc.TaskScheduleImpl.runTask(TaskScheduleImpl.java:395)
            at noticecast.scheduling.nc.NCSchedulable.runTask(NCSchedulable.java:67)
            at noticecast.scheduling.core.Scheduler.runTasks(Scheduler.java:362)
            at noticecast.scheduling.core.Scheduler.run(Scheduler.java:274)
            at java.lang.Thread.run(Unknown Source)

[SomeClown]

We've restarted services, re-created jobs, re-scheduled, eliminated possible conflicts.  The only thing we haven't done is update the App server and first dispatcher to fix pack 4.  (This was all done before my time.) 

Any thoughts? 

This.  Never intermix service packs - all servers should be at the same level of FP.  You get unpredictable results.  Until they are at the same fp, there's probably not much that can be done to troubleshoot.  (intermixed fixpack deployments are not tested)

If you still get errors, you may want to look at the java versions and get those to match - sounds like you have java installed and updated on just the one server (java_home exists?)

[jriley]

Thank you for the reply.  We had been told by IBM to make sure all are at same fix pack level with our trouble ticket as well.  The thing that perplexes me is that we will go weeks on end with everything running brilliantly and then we will have a week where they will not run at all and I need to kick them off manually. 

Also, I only see Java version from command prompt (java -version) on the App Server.  I see JRE folder beneath bin folder in install directory.  Folder titled 1.5.0 there.  I do not see Java in the Path however.   It does look like Java was updated during network server maintenance a week or so back.  Modified date in Add/Remove Programs which matches our maintenance downtime for this quarter.  So that might explain something.

Also, I was inaccurate in my description of our environment.  Our AppServer is at Fix Pack 3 along with one of the dispatchers.  The other dispatcher is at FP4.  So we'd have to update 2 of the 3. 

Our next step will be to set some down time and put this in place.  And to check into Java version mismatch.  Will follow up if any issues.  At the very least, if we still have troubles with this after, IBM support will escalate.

I'm not a dedicated Cognos admin (we have none) but more of a glorified developer who inherited these tasks so thanks again for taking the time out to reply. 

Joe