I have been working a lot more with SharePoint search lately and have been enjoying it for the most part.  Except recently when I started seeing a problem where my custom site columns were not appearing in the list of available Metadata Properties in Search Administration.

After a great deal of research and experimentation, the problem turned out to be that the standard SharePoint Http Response Header was either removed from IIS or was never added.  In either case, adding this header back in let the crawler in to see all of the appropriate metadata

Name : MicrosoftSharePointTeamServices
Value : 12.0.0.6341

This is set in Start –> Control Panel –> Administrative Tools –> IIS Manager –> [Your Server] –> Sites –> [Your SharePoint Site]  -> HTTP Response Headers

Once I had figured out the answer, I started looking for more information about why that header seems so important.  Nothing.  I did discover that if you search only the Microsoft site for this header, you will find all sorts of SharePoint related mischief when it is missing, from bad Urls to authentication errors and beyond.

My guess is that the crawl uses this to determine how it should attempt to index a site.  If it sees this header then it crawls it as a SharePoint site (including all metadata) but if it doesn’t, it then treats the site as a standard web site.  Experimentation bears this out.  I created a test case where my primary navigation control was not visible during a crawl.  If I remove the header, the index can only see  the links directly on the home page, but if I add the header and do not change anything else, it crawls everything in the SharePoint site, even though there is no navigation pointing to those pages.

Some detailed information from Microsoft about this header would be nice as I know that some security professionals advocate stripping all http response headers out of websites in order to minimize surface area. I can easily see a situation where this condition would be created.