in

SharePoint Blogs

The Best Place for SharePoint-related Blogs

Dave Wollerman's SharePoint Blog

I use this blog to share information with the community plus also as a repository for reference material on unique situations that I have come across.

View Dave Wollerman's profile on LinkedIn      Sogeti Logo

Huge MOSS Workflow Issue... What is Microsoft Thinking!!!!

There is a problem with the workflow in MOSS. For some reason Microsoft decided that they need to purge the workflow associations after they are 60 days old. This means that when you looks at the workflow history for an item you will not see workflow history older than 60 days. The information about the workflow is still in the tasks and workflow history lists, but the association between the workflow and the item has been purged from the database. I know of some people who have already contacted Microsoft about this problem and created support cases for it. The answer they keep getting is that it is by design for performance reasons. What about auditing reasons Microsoft, don't you think this will be a huge issue for enterprise clients?

 

My question is how many clients out there are using this for important business processes thinking their audits will be covered? If you are a Microsoft support customer and are using workflow for important business process that will be audited. PLEASE CALL THEM AND COMPLAIN!!! The only way it seems that Microsoft will move on this is if they get a large enough response from the community. Obviously a couple large enterprise companies (50,000+ employees) aren't enough to concern them.

 

Below are some details of the issue and some correspondence with Microsoft support...

  • Document losing their “approval workflow details” after 60 days.
     
  • Workflow details still on site, but no longer associated to respective documents.
     
  • Issue identified by Microsoft and not listed as a bug as originally told – this expiration of 60 days is something that was done purposely - “by-design”.
     
  • MS provided code that will allow us to extend the document/workflow history relationship past 60 days. (code needs to run everyday to change new associations)
     
  • Testing has not been done by Microsoft for the provided solution.

Here is a list of findings and concerns on this situation:

  • FINDING - A table that contains associations between workflow details and documents contain this expiration date of 60 days.
     
  • FINDING - This field can be changed, on a site basis, to a maximum of 9999 days – 27 years or so… at least according to Microsoft (untested as of yet).
     
  • PROBLEM – If code works successfully, it will have to be ran constantly. This must be done as new workflows are completed on documents or risk losing the associations after 60 days.
     
  • PROBLEM – This impacts all sites in a content database as they will all use this association table. I foresee a problem with maintaining separate expiration times for different sites.  This will now become the responsibility of the operations group to maintain, based the respective customer. If it doesn’t retain this permanently or semi-permanently, then this should be a setting that is configurable at a site or library level, providing the customer the ability to set the expiration period.
     
  • CONCERN – From audit perspective, Sarbanes or other, this is totally unacceptable. NO WARNING to indicate that critical detailed information will be disassociated with its counterpart document after 60 days. Unwary users could be audited 12 months after a document went thru a workflow approval and be none the wiser that their document actually was approved and their workflow history was “cleaned” 10 months earlier…
     
  • CONCERN – One of the biggest selling points for MOSS was the ability to have enhanced workflow on your documents. Almost any company is going to be negatively impacted by this “feature”. Not only financial or healthcare companies, but any financial department within a company is going to be impacted if they use this feature in MOSS. I can’t question the DOD5015 compliance as I do not know if proof of approval history is a requirement. If it is, it certainly is not DOD5015 compliant. 
  

Comments

 

AutoSponge said:

This comes as a shock.  I don't have any test histories that are more than 60-days old but I'll take your word for it.  This will certainly NOT help my case for adopting SharePoint in our compliance office.

Please keep us posted on what you find out.

September 21, 2007 1:02 PM
 

SharepointUser said:

We had the same issue and it impacted us alot and wasted alot of our time and resources. We restored our database, called Micro$oft and spent over $250 and finally and they can't find out the problem!

We found the solution ourselves by figuring out that there is a  "Workflow Auto Cleanup " option available which is available under the MOSS 2007 Central Administration located at "Central Administration > Operations > Timer Job Definitions    

Timer Job Definitions > Workflow Auto Cleanup"

Microsoft stupidly set the option to have the workflows to be deleted after a certain time. By disabling that option, we can aviod our workflow history being deleted. Thank you Microsoft! DUH!

September 21, 2007 1:15 PM
 

dwollerman said:

Awesome, that is good to know, thanks for the tip. Its too bad Microsoft can't offer up that information.

September 21, 2007 1:20 PM
 

Anil K Sharma said:

Such design assumptions should have been documented by Microsoft !!

September 23, 2007 11:30 PM
 

Karthik said:

How do we know "Workflow Auto Cleanup" job is only clearing workflow history and anything else. Is disabling that job the right solution?

September 24, 2007 2:07 PM
 

dwollerman said:

thats the only solution right now. there is no other configuration options. Supposedly Microsoft is supplying a script to auto update the number of days for the cleanup, but thats not a solution either, just a band-aid.

September 24, 2007 2:19 PM
 

Tort said:

Before you guys are so quick to bash Microsoft think about the implications of not creating this job.  Eventually the "Workflow Tasks" list and the "Workflow History" list would grow very large and performance would suffer.  This would also cause major problems eventually and also more support calls for Microsoft.  But you guys are all such big MOSS pro's that you probably knew that already...right?  The ideal solution would be for you to create your own timer job that solves *YOUR* specific needs around compliance and disable the automatic cleanup job that comes OOTB.

September 24, 2007 5:47 PM
 

dwollerman said:

It is not that there is a cleanup process as much as it was not documented by Microsoft that such a thing happens. There are people that rely on the workflow history to prove out their audits. If this knowledge was available then companies might not have 1.) purchased MOSS or 2.) bought a thrid party that handles workflow better that may or may not work with MOSS and this is where Microsoft dropped the ball.  Microsoft purposly did not say anything about this cleanup. They were quick to let everyone know all the benefits of workflow within MOSS though.

The other problem is that Microsoft knows the performance limitations of a sharepoint list. Again ball dropped. They could of easily created another database for workflows or be able to offline / archive workflow history to another storage model. Where is the workflow history for workflows not run in MOSS? Why is running workflows in MOSS different then running workflows outside of MOSS?

The reason this post is here and the reason of such a dramatic title is because it needs to get peoples attention. Companies need to be aware of this implication. It is nice that there is a "workaround", but alot of the compaines that implement MOSS for the easy workflow can't afford to have someone create a specific job for them. It would the same as if Microsoft implemented the recycle bin without configurations of a time / size limit and made people write something custom to configure it.

September 25, 2007 7:04 AM
 

Mirrored Blogs said:

Body: I was reading an article on WWF vs. LiveLink Workflow and reminiscing with my time working with

September 25, 2007 8:01 AM
 

Zlatan's Blog said:

Here's a list of some very useful updates for Windows Workflow Foundation that some of you might

October 6, 2007 5:44 PM
 

Owner Blog said:

Here's a list of some very useful updates for Windows Workflow Foundation that some of you might

October 6, 2007 7:37 PM
 

Anonymous Coward said:

Could you provide more details about the workflow cleanup?  Please name the tables in the content-database which are affected and which stored procedures are used to do the cleanup.  Does this cleanup affect the "Workflow History" list or just the "Workflow" and "Workflow Association" tables in the content-database?

October 15, 2007 2:13 PM
 

Anonymous Coward said:

I found my answer here:  forums.microsoft.com/.../ShowPost.aspx

October 15, 2007 3:01 PM
 

Robert Bogue [MVP] said:

October 16, 2007 7:38 AM
 

Robert Bogue [MVP] said:

October 16, 2007 10:38 PM
 

Andy Burns said:

Yup, I've been saying for a while that workflow Audit in SharePoint is not adequate.

In response to Tort - well, yes, I can see why they've written it this way - it fits with the whole list and items idea - but it's also pretty clear that what was written isn't up to many businesses needs.

Would a database table be too much to ask? Or to allow tracking providers to be plugged in, a la WF?

I'd love to have a clear week - I'd write a better auditing system for workflow.

www.novolocus.com/display.php

October 19, 2007 8:18 AM
 

Dave Wollerman's SharePoint Blog said:

Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! the original post Ok, recently I came across

November 21, 2007 9:14 AM
 

Kenneth Scott said:

Thanks for sharing this information.   Very surprising.

November 27, 2007 8:27 AM
 

novolocus.com » Auto-Cleanup of Workflows in MOSS said:

Pingback from  novolocus.com » Auto-Cleanup of Workflows in MOSS

January 9, 2008 8:53 AM
 

Karuana said:

Thanx to SharePoint User for documenting how to fix this.  We've been using an InfoPath form workflow for some transactions as our pilot project and recently I noticed that information "disappearing".  I just wanted them to TELL me this and not have to a) stress something was wrong b) get stonewalled by tech support, c) stress out my vendor with this repeated question and d) be able to configure it!  Sounds reasonable to me right?  Thanx for the help!

March 30, 2008 12:43 AM
 

Harmen said:

Hi,

maybe this might be of some help: I created a job on the SQL-Server that executes every night this code:

update WorkflowAssociation set AutoCleanupDays = 9999

Solved my problem.

Another question: Somebody has managed to restore the the disappeared workflow information. I found a way by inserting data into the Workflow table, but this takes an awful lot of time.

April 24, 2008 2:54 AM
 

dwollerman said:

Microsoft does not support direct modifications of the SharePoint databases. I do not agree that modifying the SQL tables as a vaild workaround. I know it is the quickest way to get this resolved, but the other options provided by Robert Bogue (www.thorprojects.com/.../708.aspx) and by Joris Poelmans (jopx.blogspot.com/.../sharepoint-workflow-history-cleanup.html) are a more supported workaround.

Keep in mind these posts were to get the community aware, which in turns make Microsoft aware of the impact of this issue. I just hope that this is rethought in the new version of SharePoint.

Thanks.

April 24, 2008 6:50 AM
 

Ramble On said:

Hi All I have found some material for you on the SharePoint limitations when hosting Workflow schedules.

April 30, 2008 8:47 AM
 

Fred Morrison said:

I just posted a sample piece of code on the MSDN forums that can be run from inside any workflow that will check and (if necessary) adjust the AutoCleanupDays value of the workflow association to the desired value.

June 3, 2008 1:59 PM
 

Andy Dang said:

It's shocking to find out from MSDN that the associated workflow history is kept upto 60 days for completed or cancelled items -> technet.microsoft.com/.../cc298800(TechNet.10).aspx . It would be good if Microsoft would inform us this feature early enough in the design because we are in the situation that our legal team needs to audit some workflow histories but it's no longer accessible from user or associated. What I've seen so far from the replies were to address how we can prevent this 60 days retention and restore backups to audit these events (also came across with unconventional way by direct access to DB). What I may after is does anyone having tried to use standard APIs to associate or provide a view to see complete audit trails? If there arent such APIs, then it would be catatrophic.

Any lights???

June 13, 2008 2:17 AM
 

dwollerman said:

I here ya, and thank you for responding, I think you are the first one to see (and is possibly living through) the issue and points I was trying to make. Yes it sucks that they are cleaned up after 60 days, but there is no warning of this.

As far as recovering the associations. As you can see there are people that have provided workarounds to help prevent the clean up, but that doesn't fix the issue if they are lost to begin with. The workflow history is still stored in sharepoint and not removed after 60 days, it is only the associations with the item the workflow was executed on that is removed. You can potentially use the standard sharepoint APIs to connect the items in the workflow history list with the workflows associated items to build a report. At this point they are both just list items.

I know that is not the friendliest way to do this and the information in the workflow history list is not formatted in the friendliest way either, but it should provide you with all the information you may need to reconnect them into a report.

June 13, 2008 7:34 AM
 

lesmantecoooo said:

Hi,

Don't you guys realize that the duration of 60 days can be changed from SPWorkflowTemplate.AutoCleanupDays property to meet your data retention requirement? Please be advised that modifying this to long duration can cause performance degradation, as historical data accumulate.

The option is there for you to change to meet your organizational need. I don't think this is a BIG DEAL. You might also want to look at Information Management Policy feature.

Cheers

July 9, 2008 1:29 PM
 

dwollerman said:

We know that this can be change when developing a custom visual studio workflow. We know that we can write custom code to run periodically to change this options as workflow is ran.

It is not configurable for sharepoint designer workflows. It is not configurable when using OOTB sharepoint workflows.

The only way it is configurable or a workaround is to have something (whether it is a workflow or a timer job or a script) developed, which most companies get frustrated with because they just spend big $$$ on the product now they have to spend more money to develop something that should be available.

The big deal part is more related to the lack of warning, documentation and explaination from Microsoft related to this "behind the scences" feature. Most companies were relieved that sharepoint finally had workflow built in, and for the most part it is a well waited for feature and works very well. The problem comes when these companies enact critical business process which require audit history and these companies rely on the associations to give this to them. I don't blame the companies for relying on this since there is no disclaimer or documentation stating that the associations are removed after 60 days.

also, i know that 90% of workflows out there don't need an extreme level of auditing and 60 days should be fine, but there are some that are required to keep it for years and companies are relying on the association for it. Good or bad, thats how it is understood. You can't blame bad consulting either, because alot of these companies walk through standing it up on their own before asking for expert advice. And they don't ask until something has already happened.

July 9, 2008 1:55 PM
 

Raevean said:

If U have to keep a workflow history list for auditing... mayby should you think about creating your own auditing activity which you can use in your workflow develop the way you wanted... I Dont really think its a big issue...

July 17, 2008 11:47 AM
 

Matthias said:

Hi,

i´ve managed to get a workaround, with 2 workflows(24h wait, look if date fits, if not kill workflow and start worflow2. this waits 24h looks etc. ;))

But theres a new problem. After a few days the Item gets an "overflow". Cant Open, Edit, view History or Workflows. I think the WF History is to much for sql, cause i get a "Unknown SQL Exception 191 occured. Additional error information from SQL Server is included below." Eventid 5586. The Statement is to deep!

Any Idea how to get around that issue?

August 13, 2008 8:15 AM
 

Jeremiah said:

August 18, 2008 8:59 PM
 

dwollerman said:

Matthias,

The issue could be with how you are "killing" the workflows. Maybe they are broken processes and the relationship between the workflow and the item was killed in the process causing sharepoint to error because it cannot find the information to join them.

August 19, 2008 10:38 AM
 

drewdevo said:

Description Our company discovered during an internal audit that workflow history had been removed from

August 27, 2008 3:02 PM

Leave a Comment

(required )  
(optional )
(required )  
Add

Need SharePoint Training? Attend a SharePoint Bootcamp!

Posts (c) their respective authors. Everything else (c) 2007 SharePoint Experts