Just thought I would post what I discovered since it took me an hour and it might help others. Basically the issue is that the most common approach to serving up custom error pages is to use customErrors in web.config which works by issuing a 302 to redirect the client to the specified error page. The problem is that this changes the status code from 404 or 500 to 302 instead, meaning that missing pages are not removed from the search engine and that missing or pages encountering an error will have the contents of the error page cached for the requested url:
404 - missing page - search engine will remove page from cache
500 - error in code - search engine will ignore page not updating it's cache for that url
302 - temporary new location (redirect) - search engine will update the cache of the original url with the contents of the new location
200 - page loaded ok - search engine will update it's cache for that url
Now ASP.NET 3.5 did introduce ...
Go to the complete details ...