Russ Houberg's SharePoint BlogUnlocking the hidden power of SharePoint for document imaging.http://www.sharepointblogs.com/rhouberg/atom.aspxCommunity Server2008-01-29T11:39:00ZMOSS Scalability and Performance WhitePaper RELEASED!http://www.sharepointblogs.com/rhouberg/archive/2008/06/14/moss-scalability-and-performance-whitepaper-released.aspx2008-06-14T14:50:00Z2008-06-14T14:50:00Z<p>Ok. So I've been waiting for this day for about 3 months now!</p>
<p>The culmination of the efforts of several incredible individuals has officially materialized. ANNOUNCING:</p>
<p><b><font size="4"><a class="" title="Microsoft Scalability Whitepaper" href="http://go.microsoft.com/fwlink/?LinkID=120891">Using Microsoft® Office SharePoint® Server to implement a large-scale content storage scenario with rapid search availability</a> </p></b></font>
<p>This document is a scalability whitepaper that has been reviewd by Microsoft and posted on TechNet. This white paper presents the key decision points, architecture design and definition, test criteria and results from a scalability and performance test conducted at the Microsoft Partner Solutions Center (MPSC) in Redmond, WA.</p>
<p>Don't be discouraged by the 97 pages of content in this document! It's an easier read than it appears. Many of the pages are graphs and charts that document the results of the test runs.</p>
<p>I just wanted to thank Andy Hopkins for breathing life into this effort and finding the funding that got it done! If he hadn't brought in KnowledgeLake, I'd still be just messing around with conversion and migration into SharePoint!</p>
<p>And I can't forget my good friend Paul Learning. This guys is amazing folks. His architecture guidance and his ability to keep us "on track" with Microsoft guidelines were excellent. Paul is a taxonomy wiz kid and a dedicated professional. I can't even count how many late nights he worked to bring this effort to life. Thanks Paul. As each of us navigate our careers, I expect to stay in touch with you all allong the way.</p>
<p>I also want to thank the SharePoint architects and product team members that consulted with us on this effort. You guys were willing to share your knowledge and I'm a better architect for it. I'm not sure if any of you read this blog but in case you do, I just wanted you to know that I appreciate that you added your knowledge to the mix. My hope is that you will continue to see me evangelizing SharePoint Scalability and Search technologies for many years to come.</p>
<p>Finally, (can you hear the Emmy exit music?) I wanted to thank KnowledgeLake. These guys continue to bless me with the opportunities that allow me to further my career and the success of KnowledgeLake. I don't take this privilege lightly. Ron Cameron, Bob Bueltmann, Gregg Smith, Chris Caplinger, and Tom Brownell. I can't imagine working for a better 5 guys. The servant leadership culture that you all live and breathe is truely appreciated.</p>
<p>So, the whitepaper is here... forever added to the ocean of information in TechNet. My hope is that it helps the community and furthers the image of SharePoint as a highly scalable collaborative and content management platform!</p>
<p><font face="Calibri" size="3">Download (.docx): </font><a href="http://go.microsoft.com/fwlink/?LinkId=120881"><font face="Calibri" size="3">http://go.microsoft.com/fwlink/?LinkId=120881</font></a><font face="Calibri" size="3"> </font></p>
<p><font face="Calibri" size="3">TechNet topic: </font><a href="http://go.microsoft.com/fwlink/?LinkId=120901"><font face="Calibri" size="3">http://go.microsoft.com/fwlink/?LinkId=120901</font></a><font face="Calibri" size="3"> </font></p>
<p><font face="Calibri" size="3">Download (.doc): </font><a href="http://go.microsoft.com/fwlink/?LinkID=120890"><font face="Calibri" size="3">http://go.microsoft.com/fwlink/?LinkID=120890</font></a><font face="Calibri" size="3"> </font></p>
<p><font face="Calibri" size="3">Download (.pdf): </font><a href="http://go.microsoft.com/fwlink/?LinkID=120891"><font face="Calibri" color="#800080" size="3">http://go.microsoft.com/fwlink/?LinkID=120891</font></a></p>
<p> </p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=17605" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxTechEd 2008 Whirlwind - RBS, FAST Search, and Hyper-Vhttp://www.sharepointblogs.com/rhouberg/archive/2008/06/14/teched-2008-whirlwind-rbs-fast-search-and-hyper-v.aspx2008-06-14T14:21:00Z2008-06-14T14:21:00Z<p>What can I say about this past week! It was a great ride and I'm not even talking about The Mummy at Universal! I spent this last week at TechEd 2008 in Orlando. It was such an amazing event that I don't even know where to begin! So I'll start with the facts.</p>
<p>Microsoft's new hypervisor technology "Hyper-V" is set to storm onto the scene. I saw some excellent demos this week. So good in fact, that I'll probably have Windows Server 2008 with Hyper-V running on my laptop and/or my desktop before the week is out. The big thing I picked up here is that the hypervisor layer has been pushed below the kernal layer so we get a very "close to the metal" connection with system RAM and disk I/O interfaces (SCSI, SATA, HBA, etc). So the old addage of losing 15-20% performance from virtualization probably goes out the window. It's probably closer to 3 - 5% now.</p>
<p>I also enjoyed talking with the enterprise search product manager from Microsoft. I look forward to talking with him again regarding how FAST is going to complete the enterprise search story for SharePoint. With any luck, KnowledgeLake will be in on the ground floor of this new frontier.</p>
<p>Lastly, Hao Zhai, a friend of mine and the Commercial Product Manager for KnowledgeLake attended a session on "Unstructured Data Storage". I totally missed this session so I'm going to plagerize the email he sent me. This single email signifies the beginning of Microsoft "getting it right" regarding large scale document repository architecture. Check this out from Hao:</p>
<p style="COLOR:red;">Went to a session today on “Unstructured Data Storage”. The presenter confirmed that Microsoft will provide a RBS (Remote Blob Storage) API for SQL 2008 that allows binary data to be stored on “remote” devices. NetApp, EMC and IBM have all officially signed up to write providers for this API, Hitachi and Fujitsu were also mentioned as “in the works”. This is in addition to the FileStream feature where blobs are stored on locally attached NTFS volumes. In both cases, SQL Server will manage the backup/link consistency between DB and blob store.</p>
<p style="COLOR:red;">The SharePoint team has committed to leverage FileStream and RBS in O14. To me, this means SharePoint will be able to truly compete head-to-head with Documentum/OnBase/FileNet in terms of storage scalability.</p>
<p style="COLOR:red;">This will replace the current “External Blob API” in SharePoint v3 SP1. The current API will continue to exist for backward compatibility for those who built solutions around it while waiting for O14. However, with the SharePoint team committing to implement a new/better solution OOTB leveraging SQL Server 2008, any investment in the current API would have very short-lived relevancy.</p>
<p>WOW! Amazing stuff! Guess what guys, without blobs in the SQL database, we will be able to take SharePoint into the BILLIONS of documents! Once remote blob storage bursts onto the scene for SharePoint even simple taxonomies will be able to scale to INSANE numbers of documents! I CAN'T WAIT!!!</p>
<p>Lastly, I want to throw a shout out to the Novell booth in the partner expo center. Thanks to them, I scored a $1000 travel voucher by winning the Tiger Woods long drive challenge in their booth! Also, I'm not sure if Rob, Khai, or Mac read this blog or not but I had fun with those guys! We all did a few round trips into the long drive line trying to one-up each others drive lengths! It was all friendly competition and I had a BLAST!</p>
<p>So there it is. With any luck we'll all soon be maximizing our massive multi-core servers using Hyper-V to run large SharePoint farms that do blob externalization all while being crawled using FAST search! So uhh, just a heads up to the Documentum guy who tried to sell his wares during my Birds of a Feather scalability session.... Might want to freshen up that resume. ;)</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=17602" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxSharePoint Database Indexes and Statisticshttp://www.sharepointblogs.com/rhouberg/archive/2008/05/16/sharepoint-database-indexes-and-statistics.aspx2008-05-16T17:20:00Z2008-05-16T17:20:00Z<p>In the storage whitepaper that I recently released, I talked a bit about the <strong>SPDatabaseStatisticsJobDefinition</strong> timer job and the fact that it can, in most cases keep SharePoint running smoothly. However, I recently had to research a bit further into what exactly this timer job does. This is what I discovered:</p>
<p>This timer job is associated with each web application in the farm. It is automatically created and configured to run weekly when the web application is created. I mentioned that it might be a good idea to run some sort of statistics refresh after a large migration (into SharePoint) or similar operation. Well, the interesting thing is that I traced this timer job definition all the way down to a stored procedure that exists in every content database. The stored procedure is called proc_UpdateStatistics. </p>
<p>So if it were my farm, I might be inclined to try to modify the schedule for this timer job but this may not be possible (haven't had time to check). The other possibility is to just execute the stored procedure manually but that might not be supported by Microsoft. Interestingly, <a class="" title="SharePoint Maitenance Plan KB Article 932744" href="http://support.microsoft.com/kb/932744/" target="_blank">according to this KB article</a>, Microsoft specifically says that it's OK to update statistics, so maybe it is supported.</p>
<p>Anyway, I just thought all of that was interesting. On a related note, I want to clarify, that this timer job runs the statistics update on CONTENT DATABASES. So it is highly likely that you will need to create an additional maintenance plan specifically for the SSP search database. If the search database is large and/or supports millions of documents, it's a good idea to keep it cleaned up regularly. </p>
<p>Finally, a bit of a warning. No matter how you manage your indexes/statistics refresh, you don't want to be updating statistics any time a crawl is running. Particularly if the crawl has a lot to do (full crawl), if you're trying to update statistics at the same time, you could end up grinding the SQL Server into oblivion.</p>
<p> </p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=16553" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxCalm Before the Stormhttp://www.sharepointblogs.com/rhouberg/archive/2008/05/06/calm-before-the-storm.aspx2008-05-06T20:29:00Z2008-05-06T20:29:00Z<p>Well, it's been a few weeks since I finished the SharePoint scalability work. The storage whitepaper I wrote is going over well with customers and the public in general and the big daddy SharePoint Scalability whitepaper will soon be posted on TechNet. Also, I'll be presenting material from the storage whitepaper at the Dallas and Houston SharePoint User Group meetings in May.</p>
<p>So it's wierd that I'm not totally slammed with work. I had kind of gotten used to it. For the moment I'm doing minor troubleshooting work and retrofiting the scalability recommendations into client environments. So with a few moments to come up for air, I decided to take stock and retool a bit.</p>
<p>First up, what's the state of web development? Man am I behind! I've had my head buried in the SharePoint sand for so long that I hadn't even heard of LINQ! And it came out like a long time ago! I ran across it while working on a timesheet application. I wanted a nice project to polish up the skills so I set my sights on a LINQ driven AJAX enabled timesheet application. The app is coming along nicely and I really like LINQ. Talk about easy! I spent a single day on the data model and BAM the data layer was done for me and I'm updating data tables. Sweet!</p>
<p>Also, I got around to taking the rest of the SharePoint MCTS exams. I've passed all 4 now! I scraped by on one of them but I thought I did pretty good for not using exam cram materials. I took the tests cold baby, with only real world experience to guide me. I'm pretty stoked that I got those out of the way.</p>
<p>Finally, I took an evening and whipped out a quick site for a <a class="" title="Harley Raffle" href="http://www.harleywinner.com/" target="_blank">Harley Raffle</a> for my church. If you live in the US and you want to throw $10 into the ring to see if you can win a Harley, <a class="" title="Harley Raffle" href="http://www.harleywinner.com/" target="_blank">check out the site here!</a> I promise, it's on the up and up. I've seen this SWEET machine in person and Frieze Harley-Davidson is holding the title and the keys to ensure it stays "new"! </p>
<p>So I don't expect the calm to last long. There are rumblings of a possible extension to the massive SharePoint scalability effort I had been working on. I hope they pan out. I love this scaling SharePoint stuff.</p>
<p>Cheers</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=16190" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxScaling SharePoint 2007 - Storage Architecturehttp://www.sharepointblogs.com/rhouberg/archive/2008/04/11/scaling-sharepoint-2007-storage-architecture.aspx2008-04-11T18:35:00Z2008-04-11T18:35:00Z<p>Ok. So after writing that last post on the 100GB database limitation I got some nice feedback from several people. Also, I'm finding that this is information that really needs to get "out there".</p>
<p>Far to often, KL is brought in to implement a document imaging or file share conversion solution in SharePoint. The problem lies in the fact that often times the sandbox that everyone learns to play in becomes production! This ends up presenting a lot of challenges for us as consultants when trying to implement solid solutions. So I want to help out where I can.</p>
<p>To that end, I've created a whitepaper that will help the SharePoint Administrator to implement a farm with a solid storage foundation. It's possible to take bits and pieces from this whitepaper in order to ensure that you've implemented a storage foundation that scales according to YOUR needs, be they small scale or large scale. Even in a smaller scale environment, there are steps we can take to optimize file I/O and SQL Server performance.</p>
<p>So I waited a while for somewhere more official to post this document. Eventually it will be up on the KnowledgeLake website and even referenced in an upcoming Microsoft SharePoint scalability whitepaper. But for now, my personal web space will have to do. This whitepaper has been done for a week now and just waiting for a home.</p>
<p>So.. Here you go world... I hope this helps!</p>
<p>Please download <a class="" title="Scaling SharePoint 2007 - Storage Architecture" href="http://www.houberg.net/blogcontent/Scaling%20SharePoint%202007%20-%20Storage%20Architecture.pdf" target="_blank">Scaling SharePoint 2007 - Storage Architecture</a> and let me know what you think! </p>
<p>It should be an easy read and I think you'll get a lot out of it!</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=15267" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxSharePoint 2007: Revenge of the 100GB Databasehttp://www.sharepointblogs.com/rhouberg/archive/2008/03/08/sharepoint-2007-revenge-of-the-100gb-database.aspx2008-03-08T16:10:00Z2008-03-08T16:10:00Z<p>Right...he's a Star Wars geek. Check.</p>
<p>I wanted to discuss something else I heard a lot about at the SharePoint 2008 conference. The 100GB database limitation.</p>
<p>Organizations are now looking at SharePoint as a legitimate large scale application. They want to believe. They want to engage. Then they all hit their heads on the same thing. 100GB database size recommendation. Folks... it's a recommendation. The answer to the question of can we go bigger is the same as what I heard several times throughout the conference... "it depends". If properly architected and with quality disaster recovery solutions in place, the content database can be larger.</p>
<p>So what I want to discuss is that the 100GB requirement is a guideline driven primary by SLA requirements. The point being that you have to be able to back up and/or restore the content databases in an amount of time that is reasonable for your business. If you're doing log shipping or have a disk to disk backup rig with an acceleration component from a Quest or Avepoint and you can nail a backup quick like, then you can go larger than 100GB! </p>
<p>The only minor performance issues that I've seen with large content database center around large list updates. For example, if you add a column or a column index to list or library that has several million content items in it then some of the data tables in the content database will be locked until the change has completed. This will effectively lock out all other users from accessing any content in that content database until the change has completed. </p>
<p>I have seen at least one content database of 400+ GB in size and I've heard of others that are about 1TB! While 1TB is definately pushing it quite a lot and performance isn't as good as with a smaller database it is usable. With a small number of users or in an archive scenario it could be acceptable. The 400+GB database runs fine. So I want to give you some tips if you are comfortable with going larger than 100GB:</p>
<ul>
<li>I/O is everything. If you know you are going to have a very large content database, then you'll do well to be generous with your storage gear.</li>
<li>RAID 5 is a minimum, RAID 10 is better</li>
<li>BEFORE you create your site collection, pre-create an empty content database. Add data files to the empty content database such that you have 1 data file for every processor "core" in your SQL Server.</li>
<li>If at all possible, place the individual files on a separate LUN or physical set of spindles</li>
<li>LUNs can be large enough to accommodate multiple data files from DIFFERENT databases</li>
<li>MONITOR the Average Disk Queue lengths of the (hopefully different) LUNs. You want to see them under 2 if possible. If you're in the decimal range then you're golden. If you're in the single digits then you're acceptable. If you see ADQ numbers into the double, triple, or quadruple digits, then you've got problems that need to be addressed.</li></ul>
<p>For example, lets say I my corporation has collected 4TB of content over the last 5 years and we want to move it all into SharePoint. For the sake of this example, we'll ignore the fact that once stored in SharePoint, the content will take up more than 4TB of space. Also, we have an 8 core SQL server with say 32GB RAM. You could possibly shuffle that content out as follows:</p>
<p>Create (8) 1TB RAID 5 or RAID 10 LUNs. Lets say we map those LUNs to drives H: through O:. Note that you could just as easily mount them to empty folders if you don't want to use drive letters. With an 8 core SQL Server and 8 content database luns, I can create 8 files per content database and put one of them on each of the different LUNs (neet how that worked out for this example!). </p>
<ul>
<li>With this rig we could pre-create 20 content databases. </li>
<li>All of the database [dbname].MDF files would be on the H: drive. </li>
<li>We then add [dbname2-7].NDF files on the i:\ through o:\ drives</li>
<li>We then create our 20 site collections probably using the "stsadm -o createsiteinnewdb" command</li></ul>
<p>We then go through the effort of getting the content into SharePoint. <ShamelessPlug>KnowledgeLake has the framework to get this done by the way.</ShameLessPlug> Once the 4TB of content is done being loaded into the 20 site collections, you will find that each content database is approximately 200GB in size. That means that each of the 8 data files for a given database is actually 25GB and spread across each of the 8 LUNs. We now have a 200GB database with excellent I/O numbers and we still have room to double in size without worrying too much about I/O performance. Of course, your mileage may vary depending on how the LUNs are configured and the performance characteristics of your SAN.</p>
<p>I want to be clear that this is a hypothetical example of one possible solution. Every organization has variables that would affect this architecture, thus fulfilling the "it depends" mantra.</p>
<p>Russ</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=13778" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxLarge Scale Architecture Questionhttp://www.sharepointblogs.com/rhouberg/archive/2008/03/08/large-scale-architecture-question.aspx2008-03-08T15:25:00Z2008-03-08T15:25:00Z<p>So the Microsoft SharePoint Conference 2008 wrapped up on Thursday. What an amazing ride! It blows me away to see how SharePoint is absolutely exploding. It fires up my passion for SharePoint technology even more!</p>
<p>Paul Learning, Andy Hopkins, and I did finally present during the last session slot of the conference. There weren't as many attendees as I would have like to have seen, but what the heck, I usually bail early on the last day too! Anyway, with the smaller group of only about 30 folks it was a less formal session. We were able to engage in some quality discussion around scalability and performance.</p>
<p>One person asked me a question that I'd like to address here. He talked about the fact that they have a large volume of files in FileNet. They'd like to move them into SharePoint but they can't come up with a way to logically group them such that they could keep the site collections/content databases inside of the 100GB recommended size. I asked him how his users accessed the content in FileNet. He didn't want to go there because the answer was that they "searched" for the content. Then he said that his users had been exposed to SharePoint and had been accustomed to just navigating straight to the documents they need.</p>
<p>So of course my answer to him was something like, use MOSS search... embrace the search... love the search...! Search is like the keys to the kingdom in a large scale MOSS implementation! He didn't want to hear that answer unfortunately. That brings me to two points. </p>
<p>First, don't expose your users to technology you don't want them to have. He feels like limiting direct navigation is like taking candy from a baby and I agree to an extent. So I encourage everyone to spend the time up front to design the system and train users up in the way that they should go from the beginning (whenever possible... I know it's hard).</p>
<p>Second, if you want to unlock the power of your MOSS implementation. You have to imerse yourself in the search capabilities of the MOSS platform. Mind what you have learned! Save you it can! Ok, enough with the Yoda references. Seriously, I get that it doesn't matter what you put in if you can't get it back out easily. But we have to break the habits of the S:\ drive. Hierarchical data structure doesn't help the new employee trying to navigate through 4TB of content!</p>
<p>Russ</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=13777" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxMicrosoft SharePoint Conference 2008http://www.sharepointblogs.com/rhouberg/archive/2008/03/05/microsoft-sharepoint-conference-2008.aspx2008-03-05T07:12:00Z2008-03-05T07:12:00Z<p>Uhhh. If you haven't gotten the memo yet folks. SharePoint is HOT!!!!</p>
<p>I've been to a few different conferences in the last couple years and it is very clear that interest in SharePoint is growing at a rapid pace. Now that SharePoint has split off from the MS Office conference there is a pure focus at the MS SharePoint Conference 2008. </p>
<p>Initial indications were compelling as the conference was sold out over a month in advance. Then the hotels sold out. Then upon arrival there were people literally hanging around outside hoping to get in. One guy commented that if we all didn't have pre-registered RFID badges there probably would have been some serious ticket scalping going on! It's not hard to see why. I haven't been around this many smart cerebral type folks in a long time. You can't help but learn something!</p>
<p>I grabbed a great session today on Planning for Scale and Capacity. So much of what they had to say rang true. Their "It Depends" answer to the inevitable "how many servers do I need" is something that I seem to have given to a lot of customers these days. Fortunately, these guys have taken a lot of the guesswork out of capacity planning with their excellent <a class="" title="Capacity Planning Tool" href="http://www.microsoft.com/downloads/details.aspx?FamilyId=DBEE0227-D4F7-48F8-85F0-E71493B2FD87&displaylang=en" target="_blank">Capacity Planning Tool</a>. They did another demo of it today and I'm enamored by how easy it is to use! It is wonderful to have such a powerful tool that takes into consideration so many variables! Great work guys!</p>
<p>Another thing I'm excited about is the story around FAST search. The message of where FAST slots into the mix is really shaping up. It's clear that you have Search Express on smaller implementations, Enterprise Search on midrange to large implementations, and then there's FAST which can do the EXTREMELY large implementations with mind boggling scales of capacity and search response times. I just don't see how any other portal product can complete with this holistic platform. They've covered all the bases. SharePoint scales to the moon. And if you have an EA with Microsoft, YOU ALREADY OWN SHAREPOINT! Calling all CIOs...lets get to work pulling together all of that intellectual content you've got scattered around! It just got even harder to come up with a reason not to!</p>
<p>Anyway, it's shaping up to be an amazing conference. I have some friends doing some sweet sessions. Darrin Bishop (my mentor from back in the 2003 days) is doing cool things with PowerShell and SharePoint administration these days and I'm looking forward to Todd Baginski's session on SSO. Unfortunately, I missed his BDC session today but I heard it was excellent.</p>
<p>Oh yeah, and what a great perk Microsoft... Free Prometric testing for the "Configuring MOSS" and "Developing MOSS" tests! I took a crack at the Configuring and knocked it out with flying colors so I'm officially an MCTS now! I'm hoping to get a chance to take the "Developing MOSS" test tomorrow.</p>
<p>Also, I want to plug the session that I've been directly involved with! Remember that whitepaper I mentioned in an earlier post? Well, Paul Learning (MS), Andy Hopkins (MS), and I have been working on this scalability effort for the last couple months and Paul and Andy will be presenting on much of what will be in that whitepaper! For any of you at the conference that happen to catch this blog from an aggregator somewhere, PLEASE don't bail on the conference early! We're one of the last sessions on Thursday! Come check out the <a class="" title="itemarrow_160" style="CURSOR:hand;" name="itemarrow_160"></a><strong>SharePoint Scalability – Practical Application for the Enterprise</strong> session at 12:00pm! It was a late entry and didn't make it into the schedule book.</p>
<p>Finally, thanks to Todd Baginski for talking us into waiting in line over 2 hours to get into the flight simulator at the Museum of Flight. It was WELL WORTH THE WAIT!. Paul and I had a blast flipping and rolling that thing like crazy. I'm looking forward to game night tomorrow! I've been away from home only 2 days now and in addition to missing the wife and kids, I'm also missing my daily dose of Guitar Hero! I hope they have it tomorrow!</p>
<p>Good times.</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=13570" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxMOSS Search Results Can Be Near Real Timehttp://www.sharepointblogs.com/rhouberg/archive/2008/02/28/moss-search-results-can-be-near-real-time.aspx2008-02-28T19:58:00Z2008-02-28T19:58:00Z<p>Well, I'm not sure how many times I can mention KnowledgeLake and "Transactional Content Management" without getting flogged by the blog hosts for peddling our wares again... but here I go again.</p>
<p>So once again, I'll set the stage with the world I work in every day. KL is all about facilitating document processing all the way from paper to grave. By grave I mean the end of a document lifecycle. So after KL Capture Server blasts a batch of documents into SharePoint we often take advantage of some form of workflow to kick off additional document/account processing. </p>
<p>For example, imagine a lending branch scanning in and releasing a series of documents related to a loan application. Upon receipt of the actual application document a workflow might be initiated. Here's where it gets interesting. During loan application processing there might be several approval steps that are based on peripheral documents such as income statements and/or loan collateral documentation. If the institution is processing many loans per day, they don't have time to wait around for an incremental crawl to take an hour or sometimes even 15 minutes.</p>
<p>So what can we do to really tighten down search result availability? Well in this type of environment I would architect the farm a certain way and setup the incremental crawl for the content source to fire literally every minute. So the information below outlines how I would configure the farm to squeeze the absolute most performance out of crawl processing.</p>
<h5>Implementation:</h5>
<ul>
<li>The farm should include a separate (and beefy) machine for Index Server. I recommend a box with at MINIMUM of 4 (64bit) CPU cores 16GB RAM running. The Query role should not be enabled on this server. Note that you can't mix 32bit and 64bit WFEs in the farm so if you're running 32bit front ends, stick with 32bit Index Server. <br /></li>
<li>In order to get that hefty Index Server to take advantage of available resources we need to force it to use more threads while crawling content. We can do that using 1 of 2 possible techniques <br /></li>
<li>OPTION 1: When configuring the "Office SharePoint Server Search" role on the Index Server, set the Indexer Performance to "Maximum":</li></ul>
<blockquote>
<p><a href="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image.png"><img style="BORDER-RIGHT:0px;BORDER-TOP:0px;BORDER-LEFT:0px;BORDER-BOTTOM:0px;" height="104" alt="image" src="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_thumb.png" width="466" border="0" /></a> </p></blockquote>
<ul>
<li>OPTION 2: We can create a crawler impact rule in Application Management => Manage search service => Crawler Impact Rules => Add Rule <br /><strong>NOTE:</strong> Crawler Impact Rules take precedence over Indexer Performance Settings and since the default simultaneous requests is based on the number of processors on the index server, it's possible that the "Maximum" indexer performance setting could be overridden by the default crawler impact setting (even if no crawler impact rules exist).</li></ul>
<blockquote>
<p><a href="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_3.png"><img style="BORDER-RIGHT:0px;BORDER-TOP:0px;BORDER-LEFT:0px;BORDER-BOTTOM:0px;" height="296" alt="image" src="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_thumb_3.png" width="624" border="0" /></a> </p></blockquote>
<ul>
<li>Then, regardless of which option is chosen, we need to set the "Target" Web Front End to be the actual Index Server itself (WFE role must be enabled) or possibly a specific "target" WFE machine would not be used for serving content to end users.</li></ul>
<blockquote>
<p><a href="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_4.png"><img style="BORDER-RIGHT:0px;BORDER-TOP:0px;BORDER-LEFT:0px;BORDER-BOTTOM:0px;" height="173" alt="image" src="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_thumb_4.png" width="623" border="0" /></a> </p></blockquote>
<ul>
<li>Finally, we set the incremental crawl schedule to fire in 1 minute increments. Navigate to the Shared Services Administration page for your SSP. Then click Search Settings => Content sources and crawl schedules => [Content Source Name]. Then click "Create[/Edit] schedule" under the Incremental Crawl field. Set the values as identified below and click OK => OK.</li></ul>
<blockquote>
<p><a href="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_5.png"><img style="BORDER-RIGHT:0px;BORDER-TOP:0px;BORDER-LEFT:0px;BORDER-BOTTOM:0px;" height="296" alt="image" src="http://www.houberg.net/blogimages/MOSSSearchResultsCanBeNearRealTime_7C17/image_thumb_5.png" width="614" border="0" /></a> </p></blockquote>
<p>That should do it. You've just configured the search service to kick off incremental crawls in 1 minute intervals! Shortly after an incremental crawl completes, if any changes were made to any of the index files, those changes will be propagated out to the Query (Search) servers. Once that propagation has been processed, the content will be available for searching!</p>
<h5>Monitoring Performance:</h5>
<ul>
<li>Keep an eye on the "Manage Content Sources" page in the SSP administration site. It will tell you the indexing status. <br /></li>
<ul>
<li>You want to watch the Indexing Status field. It will cay "Crawling Incremental" when it's crawling. It should say "Idle" when it is finished crawling. Refresh often to ensure that at some point during the 1 minute interval it is able to finish the incremental crawl. <br /></li>
<li>If Index Status never changes to Idle then unfortunately you don't have the horsepower to maintain a 1 minute incremental crawl interval. You should increase the interval by 1 minute until you verify that your crawl can complete in the allotted amount of time. <br /></li></ul>
<li>Keep an eye on the performance of your Index Server, Target Server (if applicable), and your SQL Server. If ramping up crawl performance has created an uncomfortable increase on system resource utilization on ANY of these servers, you can either back down the crawl threads (Crawler Impact Rules/Indexer Performance) or you can increase the incremental crawl duration or both.</li></ul>
<h5>Additional Points of Interest:</h5>
<ul>
<li>There are many factors related to crawl performance. Everything from how powerful your Index, Target, and SQL Servers are to the I/O performance of the SQL Server databases. The SSP Search database is particularly vulnerable as it can become very large quickly. <br /></li>
<li>Not all environments are the same. Your mileage may vary. For example, KnowledgeLake solutions often revolve in high volumes of TIFF files. There is no TIFF iFilter available for MOSS out of the box so the "NULL" iFilter is used. This means that the document metadata is gathered and inserted into the property store in the SSP Search database but the actual binary file doesn't have to be parsed. So our indexing speed is often much faster. <br /></li>
<li>With such a high load created on the Index Server and SQL Server during crawl processing, it's recommended that any Full Crawls be scheduled during off peak times (evenings and weekends, etc). This is because the Full Crawl will obey the same threading rules used by the incremental crawl. This could yield a very high level of stress on the SQL Server over an extended period of time.</li></ul>
<p>OK. That's about all I have to say about that. Once again, the cool thing about SharePoint is that it is so configurable! If the changes I specified here don't work for you, please don't flame me :) ! Just back off of the threading or put the settings back where they started and you'll be just fine.</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=13289" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxWindows Server 2003 Update Breaks RDP?http://www.sharepointblogs.com/rhouberg/archive/2008/02/11/windows-server-2003-update-breaks-rdp.aspx2008-02-11T14:01:00Z2008-02-11T14:01:00Z<p>It seems that one of the Windows Server 2003 Updates manages to hose up our ability to remote desktop (RDP) into that server. I've seen this happen several times now and never found a clean fix for it through internet research so I spent several hours one day trying to figure out how to overcome the problem.</p>
<p>First of all, there are many things that can impact remote desktop. Often times the problem is related to a hardware router/firewall, other software firewall, or Windows Firewall. Windows Firewall isn't enabled by default in Win Server 2003 but it is in Windows XP/Vista. You can test to see if Windows Firewall is causing the problem pretty easily by disabling the Windows Firewall service in Service Manager and rebooting the box. If you still can't get in, leave the firewall disabled until you fix the problem. </p>
<p>So with that out of the way, this tip falls more into the category of "I didn't change anything and all the sudden RDP isn't working".</p>
<p>I can't pinpoint which update causes the problem, but I believe I know what it does to break things. It appears that the binding of the RDP protocol to the network adapters on the server become broken after the update. In order to fix the problem, follow this procedure:</p>
<p>Start by running the Terminal Services Configuration tool.</p>
<ul>
<li>Click on the Connections "folder"</li>
<li>Right click the RDP-TCP connection and select properties</li>
<li>Select Network Adapter tab</li>
<li>Change "All network adapters..." to the network adapter bound to the IP address that you use for RDP. If it's already associated directly to that network adapter, then change to "All network adapters..."</li>
<li>Click OK</li>
<li>Reboot the server</li></ul>
<p>I've found that if I follow this procedure after losing RDP to a Windows Server 2003 update, it works every time. By the way, this can all be done in a WMI script remotely if you've got the skills for that. I'm not a WMI script guru by any stretch but I was able to figure it out the proper code in about an hour.</p>
<p>There are certainly other obstacles that can cause problems with RDP, but this is a big one that I don't think many people realize. Hope this helps somebody.</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=12543" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxSharePoint Scalability Whitepaper at SP Conference 08http://www.sharepointblogs.com/rhouberg/archive/2008/02/08/sharepoint-scalability-whitepaper-at-sp-conference-08.aspx2008-02-09T05:45:00Z2008-02-09T05:45:00Z<p>Well, the last couple weeks have been a whirlwind.</p>
<p>In the past few months, I've been working hard with Paul Learning (Microsoft Consulting Services), Andy Hopkins (Technical Development Manager, Microsoft) as well as a few guys on the MS SharePoint Product team. Paul and I have been busy loading up a massive Fujitsu server farm while Andy burns through logistical hurdles all in an effort to develop a SharePoint Scalability whitepaper. It looks like we're going to have this whitepaper done in time for SP Conference 08! So I thought I would provide a little background.</p>
<p>It's been a rediculous ride. We started with an incredible hardware rig from Fujitsu, complete with blade servers, rack servers, an Itanium SQL box and a full 10TB of storage space on a Fujitsu Eternus SAN. My job was to use some of the KnowledgeLake secret sauce and a little creative multithreading to blast 50 million records into SharePoint. Here's a hint as to the scalability of SharePoint... We were able to use the load tool to send the 50 million documents into SharePoint at a peak rate of 7+ MILLION documents PER DAY!</p>
<p>Anyway, word has leaked out that we were working on this whitepaper so I thought I might as well blog about it. In the last month or so, we've had a lot of folks contacting Microsoft and KnowledgeLake about scalability architecture and possibly executing similar tests "on their hardware". It's amazing how much buzz this is getting and we haven't even advertised it! </p>
<p>So the whitepaper is coming soon! There will be a big splash at Microsoft SharePoint Conference '08 and probably a couple webcasts after that. We're here to sing the story. SharePoint CAN SCALE, and we can prove it!</p>
<p>This is the real reason I wanted to start this blog. I wanted a forum to talk about the incredible scalability story that SharePoint has to offer. I'm a believer and I'm here to make you one too!</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=12493" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxTodd Baginski and His Excellent SharePoint Dev Coursehttp://www.sharepointblogs.com/rhouberg/archive/2008/02/08/todd-baginski-and-his-excellent-sharepoint-dev-course.aspx2008-02-09T05:22:00Z2008-02-09T05:22:00Z<p>Just wanted to give a shout out to Todd Baginski and his excellent <a class="" title="SharePoint Development Bootcamp" href="http://www.sharepointbootcamp.com/sharepoint_development_bootcamp.htm">SharePoint Development Bootcamp</a> training course. I'm sure many of you that cruise SharePoint Blogs are quite familiar with him. But for you Google Searchers (or Live Searchers Paul ;) ) who are looking for a review of his class, here's the bottom line. If you haven't had a chance to attend one of his classes or at least one of his conference lectures, you're missing out.</p>
<p>I've been writing code for about 14 years now and I've been around the SharePoint block since late 2003. I've been to many classes and lectures on various topics in the SharePoint space. I've endured several that have been light on content and full of fluff with maybe one or two interesting points. Not in Todd's class! This guy has a deep understanding of the nuts and bolts that make up the collective SharePoint API and it was a pleasure to listen to him (even though business called me out of a few sessions)!</p>
<p>Also, he left behind a collective of resources that are simply amazing. Added to my personal knowledge base is a collection of easily referenceable courseware chapters, code snippets, and various handy SharePoint utilities. Good form Todd. Good form.</p>
<p>On a personal note, I'd like to say something else about Todd. For all the knowledge of SharePoint development concepts that he possesses, there isn't even a hint of ego. It's easy for confidence in one's abilities to bleed over into arrogance and that goes for any walk of life. It's not a problem that Todd has. He's a very humble person high on character and passionate about teaching others what he knows. I really respect him for that.</p>
<p>Thanks Todd. It was a pleasure. See you at the MS SharePoint Conference '08.</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=12492" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxJoel Oleson and the Anatomy of Indexinghttp://www.sharepointblogs.com/rhouberg/archive/2008/01/29/joel-oleson-and-the-anatomy-of-indexing.aspx2008-01-29T18:45:00Z2008-01-29T18:45:00Z<p>Ok. So the real reason that I chose today to on-ramp this blog. I wanted to add a little something to Joel Oleson's post yesterday regarding the <a class="" title="Anatomy of Indexing" href="http://blogs.msdn.com/joelo/archive/2008/01/28/anatomy-of-indexing.aspx" target="_blank">Anatomy of Indexing</a>. </p>
<p>First of all, I'm sure anyone interested in my take on SharePoint is probably well aware of Joel Oleson. Most of the SharePoint community, myself included, holds Joel in the highest respect. I respect him for his wealth of SharePoint knowledge and his willingness to share it. I met him recently and in addition to being a SharePoint guru, I found him to be a very likable guy in general. Rock on Joel.</p>
<p>So about the Anatomy of Indexing. It is a great read for anyone who's interested in that black box called SharePoint Indexing. Please make sure you've read it before continuing.... Ok, I have just one little point I'd like to add from the document repository front.</p>
<p>During an incremental crawl, the call to the sitedata.asmx yields a result set of ALL the entries in the change log. This is particularly important to KnowledgeLake as we tend to blast a lot of content into SharePoint in a short period of time (like during migration/conversion operations). But this might also apply to standard SharePoint restore operations. If you find yourself performing some type of action that will cause a high number of document changes all at once, all crawl schedules (full and incremental) should be disabled if at all possible. If it is not possible to disable them, then the url path to the library where content is being loaded/restored should be EXCLUDED from the crawl. </p>
<p>If this guidance is not followed, then the call to the sitedata.asmx web service will likely time out due to the sheer volume of data being packaged and shipped via XML (fat). I've experienced this phenomenon first hand. You end up with crawls that grind themselves into oblivion and yield a whole lot of unfriendly errors in the crawl log. Once the load/restore operation is complete a full crawl should be executed. Also, if production won't be impacted, it's also a good time to do a complete index reset.</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=12063" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxMicrosoft Gets FAST!http://www.sharepointblogs.com/rhouberg/archive/2008/01/29/microsoft-gets-fast.aspx2008-01-29T18:29:00Z2008-01-29T18:29:00Z<p>Before I continue on with any other posts on this blog, I have to lay some very exciting groundwork.</p>
<p>I said in my bio that SharePoint can be architected to scale from a single server installation that handles just a handful of static documents to extremely large implementations consisting of multiple farms that can handle 50 terabytes or more. </p>
<p>Well, this is statement requires a bit of qualification. I have been part of architecture teams that have loaded 10s of terabytes into SharePoint. This is absolutely possible and I have actually developed a content load utility that can blast 50 million documents into a properly architected SharePoint system in just a couple weeks. But the problem isn't putting data into SharePoint. It's making sure you can get it back out.</p>
<p>I've always been concerned about the 50 million document limitation for an index server in a SharePoint farm. Sure you can have multiple farms. And with the MS Search Server 2008 we can even do federated searches to aggregate results. But the problem comes with relevance. I can certainly write code that kicks off searches on multiple farms and aggregates the results but I lose relevance. Without the internal ranking engines talking to each other, all we get is a group of unranked links munged together in some sort of list.</p>
<p>Enter FAST. With the <a class="" title="FAST Acquisition Press Release" href="http://www.microsoft.com/presspass/press/2008/jan08/01-08FastSearchPR.mspx" target="_blank">recent acquisition offer for FAST</a>, things may change a bit. I listened to the tele-conference call. They didn't want to come right out and talk about the implications that FAST would have on SharePoint, but I think it's pretty obvious. Heck, SharePoint has been <a class="" title="FAST Connector for SharePoint" href="http://www.microsoft.com/presspass/press/2007/jul07/07-17MSFASTSharepointPR.mspx" target="_blank">playing nice with FAST</a> since July of last year! All the sudden, SharePoint is a player in a much LARGER market. My dream is that FAST technology be baked into SharePoint or at least so closely aligned that we won't be worrying about millions of documents anymore. We'll be talking about BILLIONS of documents.</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=12062" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspxIt Beginshttp://www.sharepointblogs.com/rhouberg/archive/2008/01/29/it-begins.aspx2008-01-29T17:39:00Z2008-01-29T17:39:00Z<p>My name is Russ Houberg (queue the Knight Rider music...OK, don't). </p>
<p>I've been serving clients as a SharePoint developer/architect for several years now. I've tinkered around with blog posts in the past but they were always broad topics. My goal is to focus this blog directly on my experiences with SharePoint. I don't claim to know it all. In fact, much of what you find here will be my take on what I've learned from others. The rest will be the results of my actual successes and failures.</p>
<p>For the last 3+ years, I've been a technical architect for a great company called KnoweldgeLake. In addition to having an amazing management team, my coworkers are an excellent group of professionals. In my role at KL, I've specialized in SharePoint architecture particularly from the point of view of scalability. I've experienced many of the documented and undocumented features and limitations of this extremely versatile platform. </p>
<p>I say platform for a reason. SharePoint in many ways is a blank slate framework on which you can build just about any web driven technology. In our case, we choose to implement it as a document imaging repository. That puts us squarely in competition with products like FileNet (IBM), Documentum, Captiva and Legato. But we have something that these guys don't have...a FREE yet fully functional storage repository called Windows SharePoint Services 3.0! So back in 2003 we used our extensive experience in the ECM industry to layer a high quality set of capture and imaging tools on top of SharePoint. Just like that, we began pushing the ECM boundaries in SharePoint.</p>
<p>So before I get flogged for touting the company wares, I do have a point. SharePoint is what you make it to be. Whether you use it as a corporate intranet, team collaboration system, report center, or a document imaging repository, there is always a common theme. We put content into the system with the expectation that it's easy to get it back out. That's where I come in. I'm all about the scalability and index/search aspects of SharePoint. If SharePoint can't provide you with what you're looking for quickly and easily, then I haven't done my job. It doesn't matter how much content you have. Whether it's just a handful of static content or 50 terabytes of content collected over the last 20 years. SharePoint can handle it with the proper architecture.</p>
<p>By the way, kudos to <a class="" title="SharePoint Experts" href="http://www.sharepointexperts.com/" target="_blank">SharePoint Experts</a> for hosting this blog site and for putting together some <a class="" title="SharePoint Administration Bootcamp" href="http://www.sharepointbootcamp.com/sharepoint_administration_bootcamp.htm" target="_blank">fantastic training</a>. I've experienced their training, it's excellent stuff! KL is having <a class="" title="Todd Baginski" href="http://www.sharepointblogs.com/members/tbaginski.aspx" target="_blank">Todd Baginski</a> come in and do the <a class="" title="SharePoint Development Bootcamp" href="http://www.sharepointbootcamp.com/sharepoint_development_bootcamp.htm" target="_blank">Development Bootcamp</a> next week. I can't wait!</p>
<p>So, keep an eye out. I hope you'll find some interesting stuff in this blog!</p>
<hr><span style="font-size:10pt;">Posted on <a href="http://www.sharepointblogs.com/">SharePoint Blogs</a></span>
<img src="http://www.sharepointblogs.com/aggbug.aspx?PostID=12054" width="1" height="1">rhouberghttp://www.sharepointblogs.com/members/rhouberg.aspx