SharePoint Blogs / SharePoint University
SharePoint Blogs and SharePoint University - all in one place!
Need SharePoint Training? Attend a SharePoint Bootcamp!

Please delete cookies related to sharepointblogs.com and sharepointu.com to resolve login issues!

Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2

Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! the original post 

Ok, recently I came across a blog post called SPWorkflowAssociation.AutoCleanupDays from Robert Bogue, a SharePoint MVP, that tries to explain away my original complaint of this process as to be not that big of a deal. I believe that he missunderstood my original posting.

My overall intention of the original post was to get the word out to the SharePoint community to make sure they are informed of this process. My goal was to get people to make sure Microsoft was aware that they were not happy with this design and that it needs to be re-designed or at the least configurable. At the very least I am glad that Robert brought this to the SharePoint product group attention. What I am not happy with is what Robert explains as their reasoning behind this little hidden gem.

The main reason for cleaning out the associations is because of a performance issue. If this is a true reason, then the product team should of reconsidered their storage model for the workflow history and associations. If anyone knows the limitations of SharePoint it would be the product team and they purposely used a SharePoint list for storage knowing those limitations and performance issues. I can't see this being their only option for a storage model and if it was, then this must of been one of those last minute features added to meet a deadline of some sort.

The second reason for saying that this is not a big deal is auditing. Robert explains that the workflow history is meant to be more of a log file then an audit trail and that MOSS has its own audit logging outside of the workflow. I can see where Robert is going, but I still don't agree. Working with large enterprise customers who are audited on a regular basis will not take that the workflow history is a log not an audit trail as an explaination. These customers will need the workflow history, comments, and everything else associated with that workflow to be part of their existing retention policies for both auditing and legal situations. MOSS Auditing will show the editing of the tasks and such, but it is not as robust as the workflow association view. Meaning that there has to be a manual association made to determine which tasks ran for a particualr document or item. Plus MOSS Auditing needs to be configured to capture these events, where workflow history is automatically there. Also, auditing is captured at a site collection level meaning you will get all workflow tasks across the entire site collection making the manual assocation more difficult, where workflow associations are for a particular document or item. Again making it easier for auditing and legal situations. In business time is money, and SharePoint is fixing that for the most part opening communication and getting tasks completed on a more timely basis, but things come up (like they always will, murphy's law).

Now, I don't want people to missunderstand me. I am not saying all workflows required this. There might be some minor approvals that do not require this kind of attention. I am stateing this because I know there are organizations out there that have developed critical business processes using SharePoint workflow and will possibly need to rely on this information in the future.

Again, I just want to make sure people are aware of this. Once the community is aware of these issues, then those people and organizations can make an educated decision as to use SharePoint workflow, turn off the auto cleanup jobs, turn on auditing, etc. An informed community and community feedback to the product team (good and bad) makes for a better product in the future. Don't get me wrong, I enjoy working with SharePoint and believe that it is a product that can change a company, like email did years ago, but with the good comes the bad and I want people to be aware of the bad as well.


Posted 11-21-2007 9:34 AM by dwollerman

Comments

Nick Schenk wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 12-11-2007 3:03 PM

Dave, you bring up good points of awareness here.  My question is this:  Is the question of the audit trail one that only applies to Microsoft or do other current solutions in the market place face the same problem?

dwollerman wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 12-11-2007 5:02 PM

I believe this only applies to Microsoft because of the storage model for the tasks in the workflows. Other products that provide workflow or some sort of business process management into SharePoint have their own databases and storage mechanisms. I am not sure if those products track each step, but I believe that it can be easliy reported from their databases. It is possible that these third parties also provide a reporting function for such things as audits.

Robert Bogue wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 03-13-2008 9:24 PM

Dave -- Here's my response thorprojects.com/.../750.aspx

Sorry it took me so long.  I didn't realize you had continued the conversation.

SharePoint wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 06-20-2008 6:15 AM

It's pretty sad if performance goes down after 2000 record entries in an sql server table, which should be able to hold millions of records which can be accessed in seconds.

RUNuts wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 06-20-2008 6:30 AM

Hmm.. I couldn't respond to a follow up post from Robert:

www.thorprojects.com/.../what-is-not-a-huge-moss-workflow-issue-take-2.aspx

on his 2nd attempt at showing that MS didn't screw up, so I'll post here...

are you kidding with this post?

For (1) if there is a definite performance hit, this is not a solution, unless you are short sited and don't care what happens in a years time.

For #2, people are refering to it loosely as auditing. For the most part, we want the functionality we should have been able to keep after 60 days. PS, not everyone bought sharepoint expecting to have to go to a programmer and pay big bucks every time we wanted something that sharepoint didn't live up to.

Last, if sharepoint gets a performance hit with 2000 items in a list, the reason for the deletion of the history references was to overcome bad programming and design and that is why the customer is suffering...

2000+ items cause performance issues. What a joke.

dwollerman wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 06-20-2008 7:16 AM

obviously I completely understand your takes on the design and the reasons why they shouldn't be removing associations, but just to be clear...

- They are just deleting the associations not the actual history, but like RUNuts said in #2 that people expect it to work and not require a programmer on staff to pull critical information out of the system

- The 2000 limit on lists are for rendering a view. A list can handle up to 10 million records if properly organized with folders. I believe that Robert is refering to the performance hit you take when SharePoint has to manage 2000+ associations for the workflows hosted in the site collections that are hosted in the content database.

-To SharePoint's point that SQL should be able to handle millions of records, is that it does. In the backend all list metadata is stored in a single table for all lists for every site collection hosted in that database.

To be fair, Microsoft probably looked at the associations to be just a log and nothing more when they were designing the feature. Microsofts either didn't realize that people would require this for audits, or this part of the feature was never sent through proper quaility assurance and it fell through the cracks, which has them realizing this based on the impact and response from the community.

RUNuts wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 06-20-2008 10:48 AM

dwollerman I think we are somewhat on the same side here, but I'll just add some stuff to my initial post..

What I don't get is this:

I understand that SQL server can handle millions of records and has fast retrieval.

I know a SP list can handle over 2000 items if you use folders to filter out what is presented, although when we purchased the software we were told that folders were a thing of the past, but thats another story...

What doesn't sit right with me is this...

So, the dbo.workflow table in SharePoint stores references to all the workflows that are in progress, I.e. 1 entry for a workflow instance. So, I'm hearing that if this table gets to over 2000 records, there will be performance issues and therefore, the clean up job cleans up items that are over 60 days since completion. Has MS heard of relational databases?

Whats going on here? What else then poor design would cause performance issues when a table gets to be over 2000 records. What are they doing, sequential searches on the tables.

Robert seems to be defending MS and thats fine. Im not a MS hater. I'm just pissed off that MS doesn't really acknowledge this that much (along with some other big issues) on their side (unless you call in and after some poking you find out it is infact a major issue they need to tackle).

Rather then the SharePoint team blogging about what is great about SharePoint, which is sometimes full of half truths, how about some blogs talking about potential issues they know about that the rest of us shoud know to.

Jeremiah wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 08-18-2008 9:00 PM
Juan Larios wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 09-26-2008 11:10 AM

IT is a problem and I think that until there is something in place, SHarePoint workflows will not suffice. I have developed a Workflow Audit feature for SharePoint Site collections. Check out my blog post where I have screen shots, and a brief explanation.

www.imaginets.com/.../workflow-audit-solution.aspx

Juan

Juan Larios wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 10-27-2008 10:42 AM

SharePoint Workflow Audit solution available for download!

www.imaginets.com/.../sharepoint-workflow-audit-solution-trial-download.aspx

Juan

r0bertsid wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 11-19-2008 10:34 AM

Juan...how does one acquire the full version?

Juan Larios wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 12-09-2008 10:19 AM

Robertsid,

  Just request one from me, email me, follow the blog site and I will forward you the full solution!

Juan

Seth Flowers wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 04-22-2009 10:15 AM

Just wanted to up this thread.

SharePoint has tons of problems. Any developer knows that.

The SharePoint marketers must be the most skilled marketers on the planet.

Kevin Yuen (knyuen) wrote re: Huge MOSS Workflow Issue... What is Microsoft Thinking!!!! - Part 2
on 05-22-2009 3:25 AM

Nice article, dwollerman! I recommend this article to anyone who is considering to use SharePoint Workflow for formal workflow processing, given its audit trail issue.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?
Need SharePoint Training? Attend a SharePoint Bootcamp!
Posts (c) their respective authors. Everything else (c) 2009 SharePoint Experts, Inc.