Each time you request a Site Collection (http://domain/) or a Site (http://domain/foo/) of your Publishing Site you get redirected to the .aspx">http://domain/Pages/<WelcomePage>.aspx. SharePoint 2007 uses the 302 header (location temporarily moved) for this purpose. Surprisingly even WSS uses the 302 header to redirect a root url to the default.aspx. In comparison ASP.NET uses an internal redirect to render the default page when the root url requested: there is no redirect in this situation.

The whole issue about the 302 headers is that the redirected locations don't get crawled by search spiders which don't follow temporarily moved pages. While it's not really an issue for intranet environments it has major impact on indexing the content of Internet-facing web sites and making them searchable using a search engine.

Looking for an answer I have researched the SharePoint runtime: SPHttpHandler, SPRequestModule and PublishingHttpModule classes. As none of these has given me a clear answer I have noticed that there are multiple references to the Redirect method present in the code which uses the 302 header as well.

To solve the issue I have designed a custom redirect HttpModule which uses 301 headers instead.

The requirements

The module must rewrite all request for a Site Collection or Site. Url's of these request might but don't have to contain trailing slash (/). Furthermore the module must distinct a WSS request from a Publishing Site / Publishing Web request. Also the module has to be aware of Variations if used by the Site Collection.

The work

Firs of all we create a new HttpModule. As we want the redirect to find place as soon as possible we will hook it up in the BeginRequest event. Furthermore we want the module to be the first one to interact with the request. As we use an external assembly we need to define it as the first element in the httpModules section of web.config.

namespace Imtech.SharePoint.Enhancement.HttpModules
{
    public class RedirectModule : IHttpModule
    {
        #region IHttpModule Members

        public void Dispose()
        { }

        public void Init(HttpApplication context)
        {
            context.BeginRequest +=
                new EventHandler(context_BeginRequest);
        }

        void context_BeginRequest(object sender, EventArgs e)
        {
            HttpApplication app = (HttpApplication)sender;
            string requestUrl = app.Request.Url.ToString();
        }

        #endregion
    }
}

Because we will need the Request url later on in quite a few places I have decided to store it in a separate variable.

The first requirement states that the module should redirect only requests for Site Collections and Sites. If the requirement wouldn't have say that the trailing slash is optional you could solve it using a simple if (requestUrl.EndsWith("/")). In our situation we will have to use a Regular Expression in order to figure out whether we need to rewrite the url or not.

Regex regEx =
   new Regex(@"^https?://.*(?<itemUrl>/[^/]+\.[^/\.]+)$");
if (regEx.IsMatch(requestUrl))
    return;

if (!requestUrl.EndsWith("/",
    StringComparison.CurrentCulture))
    requestUrl += "/";

If the url matches the regular expression it means it's a page request and should be passed on along the request pipeline unaltered. Later in the module we will combine the request url with the page url. As the trailing slash is optional I have decided to add it at the end if not present - just to be sure that combining the destination url of different parts will produce correct result.

The next requirement is distinction between WSS and Publishing Site requests.

string destinationUrl = String.Empty;

SPSecurity.RunWithElevatedPrivileges(delegate()
{
    try
    {
        using (SPSite site = new SPSite(requestUrl))
        {
            using (SPWeb web = site.OpenWeb())
            {
                if (PublishingWeb.IsPublishingWeb(web))
                    destinationUrl = String.Concat(requestUrl,
                          publishingWeb.DefaultPage.Url);
                else
                    destinationUrl = String.Concat(requestUrl,
                          "default.aspx");
            }
        }
    }
    catch { }
});

Based on the request url we create a new instance of SPSite and then open the requested web. As we can fail at this point already (for example when passing a list url) I have decided to catch the thrown exception to avoid turning the request into an error message. The distinction itself is quite straight forward and makes use of the IsPublishingWeb method. One important thing: because we are very likely to use the module for anonymous users we need to run the code with elevated privileges: the IsPublishingWeb method requires some extra permission in order to run.

Our last requirement was making the redirect module aware of Variations if used by the Site Collection. Depending on the requirements defined by your customer you might need to implement the standard SharePoint Variation logic which chooses the variation basin on the User Agent language settings. Unfortunately most users are not aware of the existence and usage possibilities of the language settings most of our customers choose to load the Dutch variation by default. If your customer requires the standard SharePoint approach you would need to implement the logic from the VariationRootLanding User Control in the ControlTemplates directory. I will focus on the scenario we're using.

PublishingWeb publishingWeb = PublishingWeb.GetPublishingWeb(web);
if (publishingWeb.DefaultPage.Url.EndsWith("/VariationRoot.aspx",
    StringComparison.CurrentCultureIgnoreCase))
{
    string defaultPage = String.Empty;
    using (SPWeb nlWeb = site.OpenWeb("nl"))
    {
        defaultPage =
           PublishingWeb.GetPublishingWeb(nlWeb).DefaultPage.Url;
    }

    destinationUrl = String.Concat(requestUrl, "nl/", defaultPage);
}
else
    destinationUrl =
        String.Concat(requestUrl, publishingWeb.DefaultPage.Url);

In most scenarios the variation redirect finds place at the Site Collection level. The default page of the root web is then set to Pages/VariationRoot.aspx. Knowing this we can check whether we need to use the variation redirect or not. The rest is quite straight-forward: we obtain the Dutch site and its Welcome Page.

The last part is the redirect itself using the 301 header:

if (!String.IsNullOrEmpty(destinationUrl))
{
    app.Response.AddHeader("Location", destinationUrl);
    app.Response.StatusCode = 301;
}

The destination url might be empty if an exception has occurred during the request processing. We will therefore redirect only if a destination url has been set by our module.

Putting it all together:

namespace Imtech.SharePoint.Enhancement.HttpModules
{
  public class RedirectModule : IHttpModule
  {
    #region IHttpModule Members

    public void Dispose()
    { }

    public void Init(HttpApplication context)
    {
      context.BeginRequest +=
        new EventHandler(context_BeginRequest);
    }

    void context_BeginRequest(object sender, EventArgs e)
    {
      HttpApplication app = (HttpApplication)sender;
      string requestUrl = app.Request.Url.ToString();
      Regex regEx =
        new Regex(@"^https?://.*(?<itemUrl>/[^/]+\.[^/\.]+)$");
      if (regEx.IsMatch(requestUrl))
        return;

      if (!requestUrl.EndsWith("/",
        StringComparison.CurrentCulture))
        requestUrl += "/";

      string destinationUrl = String.Empty;

      SPSecurity.RunWithElevatedPrivileges(delegate()
      {
        try
        {
          using (SPSite site = new SPSite(requestUrl))
          {
            using (SPWeb web = site.OpenWeb())
            {
              if (PublishingWeb.IsPublishingWeb(web))
              {
                PublishingWeb publishingWeb =
                  PublishingWeb.GetPublishingWeb(web);
                if (publishingWeb.DefaultPage.Url.
                  EndsWith("/VariationRoot.aspx",
                  StringComparison.CurrentCultureIgnoreCase))
                {
                  string defaultPage = String.Empty;
                  using (SPWeb nlWeb = site.OpenWeb("nl"))
                  {
                    defaultPage =
                       PublishingWeb.GetPublishingWeb(nlWeb)
                       .DefaultPage.Url;
                  }

                  destinationUrl = String.Concat(requestUrl,
                    "nl/", defaultPage);
                }
                else
                  destinationUrl = String.Concat(requestUrl,
                    publishingWeb.DefaultPage.Url);
              }
              else
                destinationUrl = String.Concat(requestUrl, "default.aspx");
            }
          }
        }
        catch { }
      });

      if (!String.IsNullOrEmpty(destinationUrl))
      {
        app.Response.AddHeader("Location", destinationUrl);
        app.Response.StatusCode = 301;
      }
    }

    #endregion
  }
}

To see it working build the project, copy the assembly to the bin directory of your web application and add the following element to the httpModules section of the web.config:

<add name="ImtechRedirectModule"
type="Imtech.SharePoint.Enhancement.HttpModules.RedirectModule" />

Summary

Redirects using the 302 header can form a serious issue on Internet-facing web sites as it comes to indexing the content of a web site. Using custom HttpModules to overrule the standard behavior of SharePoint is a flexible solution for this challenge.
The example above should work good enough in most scenarios. Depending on the requirements of your customers you might need to extend it with some extra functionality like for example standard Variations logic support. Custom HttpModules prove the extensibility and flexibility of SharePoint 2007 and the way it can be made to fit various requirements and scenarios.