Getting error "Job found, but wrong state"

[Former Member]

We upgraded from IdM SP3 to SP5 a few weeks ago, and (I believe it is only since the update) we have begun to see this error popping up periodically in our system log, with no other information given.

"Job found, but wrong state"

I'm not 100% sure it was the update itself either, as we transported some of our own enhancements at the same time. It never seems to be on the same job, and in fact appears to happen completely randomly, which is making reproducing it incredibly difficult. A job can run 30 times in an hour and the first 10 times it will run fine, then it will get this error once, and then it will run fine again 19 more times (with no apparent delay after the error, either).

It looks like this error is coming from the stored proc "mc_getx_job", when "mc_job_set_start" returns a Pstatus besides 0. Investigating that stored proc, I can't immediately see why it would be returning a bad status. Everything looks OK...

Has anyone else run into this problem?

[Former Member]

Hi Guys,

I am having the same issue, th error "Job found, but wrong state" keeps popping up randomly.

Did you find a solution yet ?

Thanks in advance.

[Former Member]

Not a complete solution, no.

If I do figure it out, I'll make sure to come back here and tell you guys what I found!

[Former Member]

Still having this problem in our production IdM system, it has not stopped... It appears to be during heavy load or possibly when the same job is triggered to be run multiple times quickly.

I'm wondering if maybe I should open a support ticket with SAP instead of just posting here, as this may be an internal IdM issue (or maybe expected behavior in some circumstances?) and it doesn't appear that anyone else here is having this problem.

[Former Member]

Bumping this because we are still having this issue on our Production instance of IdM, and it is causing random jobs to fail at random times, which seems like a very bad thing. I'm crossing my fingers and hoping someone else has had this issue and resolved it.

[former_member2987]

Adam,

Are your dispatchers all on the same host or on separate hosts?

Matt

[Former Member]

For our Production system, as I posted above:

We have one runtime server with two provisioning dispatchers and one regular dispatcher.

Our Test system is a little bit different. It has two runtime servers, each with one provisioning dispatcher and one regular dispatcher. The problem appears to occur in the same way with both setups.

[former_member2987]

Yep, got it. You need to get those dispatchers on separate servers. One dispatcher per server is the best practice.

[Former Member]

Yeah, looks like that was it. I've gone through and set all of the jobs that were having problems to only use one dispatcher for the time being (until we get our second VM instance set up for the second dispatcher to run on).

[paul_abrahamson_sap]

We've just exported and imported our tasks and jobs from a sandpit system to our development environment. In Development we have now started seeing this error on a couple of jobs.

We have one runtime with a dispatcher on the same server asv-idmdev, and two further provisioning dispatchers on two separate servers in another domain. These dispatches (cfs-rt1 and cfs-rt2) will be used for the provisioning to our active directory and the calling of PowerShell scripts.

In our Sandpit environment the configuration was the same with exception of only having one dispatcher in the AD domain.

On the jobs that are failing, only one dispatcher (cfs-rt1) is selected for running the job.

Edited by: Paul Abrahamson on Nov 15, 2010 12:14 PM

[Former Member]

We removed all but one dispatcher from the job definitions of all of the jobs that were being affected, and while it seemed at first to help, we're still seeing these errors. I really thought we had it nailed there... Back to the drawing board, I guess. There are now jobs with only one dispatcher marked that are getting this error.

Paul: Welcome to the club. I'll continue to post here if I discover anything new.

Edit: I just thought to check whether maybe the same one dispatcher was installed and running on multiple hosts (I imagine this would break things?) but it was not. You might want to check this also, though I don't know if it will help since it didn't solve the problem for me...

[paul_abrahamson_sap]

No we only have the dispatcher running on the one host.

Incidentally it's running on a Windows 2008 VM in order to execute PowerShell commands. However we're now having problems with PowerShell 64 bit vs 32 bit. As a result we may go back to running the dispatcher on a Windows 2003 VM, which may resolve this job found, but wrong state issue. Will update if this has any effect.

[former_member2987]

Adam,

Are all of the jobs on the same dispatcher?

Are you using multiple dispatchers? If so, are they on the same host?

Have you tried creating a separate dispatcher on a different host, and assing some / all of the problem jobs there?

Just some ideas,

Matt

[Former Member]

We have one runtime server with two provisioning dispatchers and one regular dispatcher. The problem is happening with jobs on both types of dispatchers, it does not seem to matter which kind or which one. I've seen far less of this error over the past few days though, so I'm hoping maybe it will just resolve itself eventually...

Edit: Still happening as recently as this morning. Darn, I was really hoping it was just working itself out.

Edited by: Adam Harwell on Sep 9, 2010 10:56 AM

[Former Member]

I compiled a list of all the jobs that were throwing this error to see if I could find a pattern. I could not.

There are 20+ jobs that have had this error at least once, several of which have had the error up to 7 times (spread out fairly evenly over a week or so, not just all at once). Some of them are custom jobs that haven't changed since 2009, some are jobs we just created last week, and some are SAP delivered jobs like "SetABAPRole&ProfileForUser".

I've yet to find a pattern in the timing either. As far as I can tell, they seem to have this error completely randomly.

I'll post again if I come up with any other information that might be useful.