URL Rewriting & Custom Error Pages In ASP.NET 2.0

Recently I’ve been working on several ASP.NET 2.0 sites and had to research the best ways to set up URL rewriting, permanent redirects, and custom error pages. ASP.NET 2.0 is a popular platform for building corporate websites.  Among my clients, this solution seems to be very popular with medium to large corporations who have an Information Technology department to run their website.

The World Wide Web uses a Hypertext Transfer Protocol (HTTP) to connect user programs (browsers) with web servers. Each HTTP request from a browser causes the server to return a response. That response includes a header, with a status code, and maybe some content such as a web page. Using the correct status codes can have a significant impact on search engine optimization. 

For instance, when you permanently move a page, if somebody goes to the old address, you want to make sure your server returns status code 301, Moved Permanently, with the new address for the page. This helps search engines and users find the new page location, and ensures that PageRank transfers over.

When deleting a page, you usually want the user to get a custom error page with your branding and navigation that leads them to useful places on your site. You also want the server to return status code 404, Not Found, so search engines delete the page from their indices.

Shari Thurow provides a good explanation when to use each of these methods in Don’t Abuse Users’ Search Experience With 301 Redirects.

How can you tell whether you have an ASP.NET 2.0 server? You can ask your hosting provider or IT department, or you can use software, such as the Web Developer extension for Firefox, to inspect the server response headers.

URL rewriting

I started my research at Google. Unfortunately for Microsoft, the second search result for “URL rewriting ASP.NET 2.0” was ASP.NET 2.0 Url Rewriting crippled to the point of uselessness. The fourth result was Matt Cutts’ post, ASP.NET 2 + url rewriting considered harmful in some cases. Not only does Microsoft have problems with their product, they also seem to have challenges with SEO and reputation management.

Microsoft’s URL rewriting technology does not inspire much confidence. Fear not. I have a workaround that produces excellent results. ISAPI_rewrite from Helicon software costs $99, and allows you to specify URL rewriting and redirects via an httpd.ini file in the root directory of your site. Free-standing configuration files like httpd.ini (or .htaccess in Apache) are very convenient because you only need FTP access to set up URL rewriting and redirects. The only downside with ISAPI_rewrite is that you have to convince your hosting provider or IT department to install the software.

ISAPI_rewrite’s httpd.ini file uses syntax that’s easy to learn if you already know .htaccess and Apache mod_rewrite. Here’s sample code:

# Page moved to new location. Return a 301 redirect
RewriteRule /oldurl.aspx /newurl.aspx [RP, I]

Here’s a more complicated example that shows how to canonicalize URLs to a standard hostname. You have to replace both instances of ‘test’ with your domain name, and you can delete the www subdomain if you prefer not to use it.

# Canonicalize URLs to use a standard hostname
RewriteCond Host: (?!^www\.test\.com$).*
RewriteRule (.+) http\://www.test.com$1 [RP,I]

ISAPI_rewrite can also convert URLs into user-friendly formats if you have an e-commerce shop or catalog.

Custom error pages

ASP.NET 2.0 offers a quick and easy way to set up custom error pages. Yes, it’s quick, easy and broken. According to Microsoft, the way to set up custom error pages for ASP.NET sites is via the Web.config file by adding code like this:

<configuration>
<system.web>
<customErrors mode="On" defaultRedirect="pageerror.aspx">
<error statusCode="404" redirect="404.aspx" />
</customErrors>
</system.web>
</configuration>

If you follow Microsoft’s advice, you’ll get a double whammy. First, when somebody requests a non-existent .aspx file from your server, they will receive a 302 temporary redirect to your error page, and then the error page will return code 200. These codes tell search engines to keep the deleted page in the index, exactly the opposite of what you want them to do!

To work around this ASP.NET deficiency, use ISAPI rewrite to create rewrite rules for the deleted pages to direct traffic to a custom 404 page called 404.aspx. This won’t work if somebody types in a bad URL, but as a practical matter we’re most interested in trapping errors caused by inbound links to deleted pages. By using wildcards, it’s easy enough to rewrite all your deleted URLs to the custom 404 page. When trapping errors, we silently rewrite the URL to avoid returning a redirect. Here’s sample rewrite code for the httpd.ini configuration file:

# remove all pages in the 2005 catalog and show custom error page
RewriteRule /catalog_2005/.* /404.aspx

In the custom error page, 404.aspx, include the following code to generate a 404 Not Found header status-code:

<script runat="server">
private void Page_Load(object sender, System.EventArgs e)
{
Response.Status = "404 Not Found";
}
</script>

When users or search engine bots follows a link to a deleted page, they will get your custom error page containing useful navigation and a 404 status code.

As with many things in SEO, details are important. When setting up redirects and custom error pages, be sure to test all realistic scenarios to make sure you deliver the proper status codes. As suggested by Cutts, the Live HTTP Headers add-on for Firework is an excellent tool for debugging redirects and error pages.

Jonathan Hochman has two computer science degrees from Yale. He runs an Internet marketing consultancy and a web development shop.