Macro Failures in Contributor Planning 8.4

[wonderkid]

Just curious, has anyone else experienced routine failures in their macros with Contributor Planning 8.4? We run a macro 6 times per day (consists of running admin links, a GTP, and publish for each of our Contributor Planning applications). The macro will fail atleast twice a week during any one of our runs. The error message varies, but usually it's resolved by kicking off the macro again or kicking off the particular step at which it failed. From what I understand, the failure rate we're seeing is uncommon. So, i'm just curious if others have had a similar issue and what if anything you did to resolve it.

[craig_karr]

Which step is failing? We had exactly the same issue that one macro step would fail quite frequently for no obvious reason and then when we reexecuted it without changing anything it went fine. In our case it was an admin link with a Cognos package as source that was failing. What we did to solve it was that we scheduled a restart of the services before the macro is executed and since then it has never failed. We only ran this macro during night time so that might not be an option for you, but are you on FP2? I think that they have corrected issues with admin links from Cognos packages in FP2 so it might be worth trying if that is the step that fails for you as well

[Extirpator]

If you have multiple job servers, there is the possibility that you are getting the deadlock error during the reconcile phase of the GTP. The FP2 (Fix Pack 2, released Jan 12, 2010) references: PK89386 "Deadlock on lock resources when running a reconcile with a large number of job processors."

If your admin links involve Cognos Packages, then FP2 references: PK88106 "Administrator Link with Cognos Package as source and SQL Native Client fails" and PK96106 "Admin links fail with FM source."

craig_karr mentioned restarting the services, which has significantly helped us reduce "instability" issues. Unfortunately this affects Contributor users (I think), so it may not be practical. I think restarting the services helps clear up the memory leak issue... which FP2 references: PM00249 "Server side memory leak."

Perhaps FP2 will help with your issue.

Since you are able to run the macro successfully by trying again, I'm favoring the deadlocking idea; however, your error message should have consistently referenced a deadlock. Without more information on your error messages, I'm just guessing.

Make sure that you don't have a virus scanner locking your files as Cognos creates them. This can cause a conflict where Cognos creates a file, the virus scanner scans the file, and Cognos is told the file is locked (in use). Depending on the timing, this could cause intermittent failures.

[wonderkid]

Thanks for the tips. In terms of which steps are failing, it runs the gamut (i.e. see failures with links, GTPs (or subsequent reconciles), and publishes). I misspoke in saying that the errors are typically resolved by re-running the step in question. It's about 60/40 (with us more often than not having to do one or two extra steps to fix the problem) .  Below are some of our most common issues:

• Issues in which a link or GTP will just continue to fail. We've found that many times when it just fails repeatedly, it has to do with some data having been left in the export queue of a particular app (for some reason the Export_Queue_Tidy won't have successfully done its job). We've created a export table truncation script to clean out this data when we run into such a problem, and this usually works, though it's frustrating that such a script is even necessary.
• An "Unable to execute macro [name of macro]~~~~~unable to execute macro step [name of step].....~~~~An error occurred inside the Planning Service" error. Sometimes the error message goes on further to say "...the call to JobEnd was not successful" or "...the call to JobBegin was not successful".
• We've also seen a cluster corruption bug which when it hits, locks up our entire job cluster, causing us to have to recycle services on one or all of our servers to get things going. I'd say that these are our biggest issues.

We're going to move forward with FP2. I'm just a little leery on how successful it'll be in solving some of the random failures we've been seeing for a while now. Given that our processes are scheduled to run 6 times per day (every few hours), and that we're a fairly large company with a complex finance organization, it's important that we achieve more stability with these macros.

Thanks again for the tips. Any others are more than welcome. I'm going to look into the virus scanner possibility as well. That has come up in the past during our internal discussions, and I think it's something worth looking into.   

[mizu]

hi there,

i had hard times with these issues as well,.. in the end, i found out, that contributor is not able to manage running more macros than it has available planning application servers with job processing enabled in cognos configuration ,.. no matter what settings i used in CAC for job servers.

but the users did not have analyst so they had to run these macros through web interface, so i decided to use this workaround:

Client (Cognos Connection Portal) -> ASP web site -> Batch file -> CepBatch.exe

Now, the client connects to the ASP web page listing all analyst macros (hardcoded) and clicks on an execute link next to a macro. The ASP script runs a batch file that runs an Analyst macro through CepBatch.exe utility, which is a component of every planning installation, except contrib web client (meanwhile, it creates some logs that are as well accessible for the client through the web).

Finaly, there were two more problems:
- it was not possible to execute more than one macro at a time through cepbatch, neither, so i created some kind of a lock - while one macro is running, users cannnot run another..
- i used a standalone server with only analyst and contributor installed (and iis of course). as i found out after tons of searching and troubleshooting, you cannot run an exe file through web interface, while no one is logged to the machine,.. by no one i mean the user having permissions to run the exe and all its components (dll's, registry entries, etc). then i found out that the solution would be to put the whole ASP thing to a server running cognos planning service and from then everything worked great :)

another thing is the configuration of IIS itself :) the main point here is you have to ensure that the whole thing is running under your cognos user.

hopefully this helped ;)

[ciaronk]

Wonderkid: Did you ever get to the bottom of this.  We have similar issues with our macros?  Two macro that run over night, that seem to randomly fail with

'An error occurred inside the Planning Service. Error Details: Exception occurred.'

We're on FP2 and still seeing this issue.

[Rutulian]

Hi Ciaron,

You can come at this a few different ways...

If you are seeing one or two macros repeatedly failing in the same place 3 out of 4 times, try running a Job Doctor against it.  This is a Contributor Macro Step that will produce an html report about the job's execution.  From this you'll be able to see which Job Server your failure occurred on, go there and look for any PlanningErrorLog.csv files which might have more details.  These often turn out to be things like one of four job servers not having a DB client installed.

If it's generally random failures across macros in different places, there's more chance it could be environmental.  The process above will get you to the basic error you've hit this time, but often in circumstances like these you'll get different failures at different times.  These are usually easier to avoid by tackling common triggers than analysing backwards from the root cause - 2 quick things to check are a) do all job servers have the same CPU:RAM ratio and b) are the job servers doing something else when the failure occurs?  I've seen this sort of behaviour when a backup tool is running at the same time as the macro and trying to back up gigs of temporary processing files, the job server fails because it can't clean up after itself.  Virus scanners can also give similar symptoms, as mentioned by an earlier poster.

Other than that, keep an eye out for any pattern you can see like the failure happening around the same kind of time, even if the macro runs at different times, or the failure always occurring on one particular machine.

Cheers,
Alexis

[jwilliamstd]

I have a similar problem. If anybody can provide some light, that would be off great help.

I have CAC Macro with 2 steps.

Step 1. With a publish from Contributor
Step 2. With a publish from Analyst.

Step 1. works fine. But when comes to Step 2. It runs for 10% and stop and it get FAILED when timeout happens.

I used the "execute command" generated by the WIZARD from the Analyst. When I use that command prompt it works fine, but pops for authentication.

So I changed the command as following = cepbatch.exe -m -1 501245111 -2 Publish_Report -3 username -4 pwd

Providing userid / pwd. Still its popping for the authentication. Anybody can help ??

[ericlfg]

Hi jwilliamstd,

From version 8.2 forward you are unable to pass authentication information through the cepbatch.exe utility.  In this case, the best practice is to use the 'execute analyst macro' step from the CAC.  This would call an Analyst macro that you have setup to publish what you need from analyst.  You can then use cognos connection, or the epMacroExecutor to schedule this if you so wish..

Hope this helps.

[mr j]

hi there,

i had hard times with these issues as well,.. in the end, i found out, that contributor is not able to manage running more macros than it has available planning application servers with job processing enabled in cognos configuration ,.. no matter what settings i used in CAC for job servers.

Hi,

does the above mean that if there's an environment with two Job Servers in a cluster defined in CAC I can only have max two CAC macros running at the same time? This surely can't be the case, can it? Could someone please confirm.