What a week! 

I have had many challenges throughout my IT carrer, but this week has to rank near the top as one of the most stressful. I decided to take an offer with a new company, and turned in my 2 week notice. I hate leaving loose ends, and definitely don't want to leave my present company in bind, so I have a few projects that I wanted to complete before I left. Well, one of those projects included creating a workflow for a custom list for one of our departments. When I saved the workflow to the server and created a new record to test it, I found that it wasn't working. "Failed on Start" was all that was showing.

I looked at the code and everything checked out so I dug a little deeper and found that NONE of the workflows were working throughout the entire portal! ARGH! What happened? I started digging through the log files and they weren't much help at all. This is the only thing that was showing for workflows:

2/2007 14:11:51.52 w3wp.exe (0x10A8)                              0x05BC         CMS                                     Publishing                        7fz4    High             WARNING: Cannot change FormContext.FormMode to [Invalid] because it is already set to [Edit] 

08/02/2007 14:11:55.05     w3wp.exe (0x10A8)                              0x0EB4         Windows SharePoint Services       Workflow Infrastructure             72fs    Unexpected  RunWorkflow: System.ArgumentException: Value does not fall within the expected range.     at Microsoft.SharePoint.Workflow.SPWorkflowActivationProperties..ctor(SPWorkflow workflow, Int32 runAsUserId, String associationData, String initiationData)     at Microsoft.SharePoint.Workflow.SPWinOEWSSService.MakeActivation(SPWorkflow workflow, SPWorkflowEvent e)     at Microsoft.SharePoint.Workflow.SPWinOeEngine.RunWorkflow(Guid trackingId, SPWorkflowHostService host, SPWorkflow workflow, Collection`1 events, TimeSpan timeOut)     at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow originalWorkflow, SPWorkflow workflow, Collection`1 events, SPRunWorkflowOptions runOptions)      

08/02/2007 14:11:55.05     w3wp.exe (0x10A8)                              0x0EB4         Windows SharePoint Services       Workflow Infrastructure             98d7   Unexpected  System.ArgumentException: Value does not fall within the expected range.     at Microsoft.SharePoint.Workflow.SPWorkflowActivationProperties..ctor(SPWorkflow workflow, Int32 runAsUserId, String associationData, String initiationData)     at Microsoft.SharePoint.Workflow.SPWinOEWSSService.MakeActivation(SPWorkflow workflow, SPWorkflowEvent e)     at Microsoft.SharePoint.Workflow.SPWinOeEngine.RunWorkflow(Guid trackingId, SPWorkflowHostService host, SPWorkflow workflow, Collection`1 events, TimeSpan timeOut)     at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow originalWorkflow, SPWorkflow workflow, Collection`1 events, SPRunWorkflowOptions runOptions)        

Well, to back track a little and set the scene, I had an issue a couple of weeks ago that caused me to have to do a restore from backup.  Upon the restore, I got my usernames for the shared services and application pools a litlte mixed up and wound up assigning the wrong username to the main portal app pool. To the best of my knowledge, they have the same permissions, but apparently one is missing something somewhere. (There's a lot of permissions to set on a farm implementation) I attempted to change it using the Central Admin interface, but to no avail. It didn't change in IIS. So, against the grain (totally taboo in the MOSS world), I tried to change the username and password for the main application pool in IIS. I clicked apply, ok, then did an IISRESET, and went back to check it at WHAM! Somehow, miraculously, it had changed itself back to the previous username and password! How is that possible? After a week of beating myself up over this, and hoping that it was somehow related to my worfklow issue, I called Microsoft Premier Support.  The fellow I spoke with was very knowledgeable and friendly, but had never seen this issue before either. We spent 3.5 hours working on it one day and another 3 hours the following day. We found that if we created a new app pool in IIS and assigned the main web application in SharePoint to that pool with the proper identity, workflows started working!! However, I digress..

That was not satisfactory as it was not supported by the MS team. So we deleted that app pool, and proceeded to delete the current web application using the Central Admin tool and recreate it, attaching the current content DB.  Once done, the workflows were broken again! We checked the IIS app pool and it was back to the wrong identity! We tried changing it manually with the same result as before, failure after IISRESET.  He put me on mute while he discussed options with colleagues because we were all bumfuzzled at that time. While I was waiting,  I started skimming through my MOSS Administrator's Companion and stumbled on a section instructing someone on how to change the username and password for an SSP. I thought to myself, "Surely, you've checked that, but let's do it again just in case..." Sure enough, there was the SSP credentials running under the same username as the app pool. I changed it to the correct one, did the same for the main app pool in IIS, did another IISRESET and BOOYAH! IT WORKED!!!!

So Ladies and Gentlemen, the morale of the story is this: If you do a restore from backup, make sure to use the same usernames and password for everything, but if you don't, change the username and pass for the main SSP, change it for IIS, and reset and you'll be back to normal. On one hand, I love SSPs for the additional load balancing functionallity, but they are truly a pain in the rear.

I hope this helps someone, if so please feel free to leave feedback/comments!

Rod