Our web servers run IIS5 and we also make use of the Microsoft URL Scan
utility: (http://www.microsoft.com/technet/security/tools/urlscan.mspx).

By default Microsoft's URL scan utility blocks a number HTTP Methods
including "HEAD". We have a number of clients concerned that blocking the
HEAD method will interfer with web crawlers (google bot and msn bot).

I can't really find any information which indicates if this is true or not.
Our sites are being listed in search engines so I suspect its a non issue
but I can't really find any official documentation to backup my opinions.

Can anyone provide any information?

Thanks in advance,
Brad

Hi Brad,

I'm not familiar with the details about how Internet search engines like
Goggle crawling on web sites. However from my point of view, they must use
GET verb instead of HEAD to get contents being indexed. So if your site can
be properly listed by Google or MSN searching, I assume they are still
working smoothly without the HEAD verb being allowed.

If your clients do have concern on are concerned with this, I think you can
just enable HEAD verb in URLScan.ini and this will not bring any additional
security risk. URLScan blocks HEAD verb due to it's not being frequently
used but not indicating it's a potential flaw.

Please let me know if you have any further questions. Thanks.

Best Regards,
WenYuan Wang
Microsoft Online Community Support
==================================================
Get notification to my posts through email? Please refer to:
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at:

http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

Brad wrote on Tue, 18 Jul 2006 20:25:00 -0400:

[quoted text, click to view]

HEAD is handy to determine if the page has changed since it was last indexed
(and as only the header information is returned, is much more bandwidth
friendly), and if so it will then request the page to index the content.
However, looking at my own IIS logs, I don't see any of the bots that are
indexing the site using it. In a site made up of only dynamic pages (eg.
ASP/ASP.Net) HEAD requests are often next to useless as the last modified
date returned is almost always the current server date/time. Blocking HEAD
requests may save server resources as those dynamic pages won't be run
unless data will be returned to the client (even with a HEAD request the
code in the page will need to be executed, it's just that IIS discards the
content that would be returned for a GET/POST request)

Dan

[quoted text, click to view]
In my life URLSCAN creates more problems for me than solutions.... what I do
instead is use the Microsoft Baseline Security Checker on the server and
follow the advice and I never use URLSCAN !!!

PS.: things only break on the server (for me) after I use that tool, so I
don't use it !

Hi Brad,

Just want to check if the issue has been resolved?
If it still persists, please don't hesitate to update here.
We'll go on to assist you on it. Thanks. :)

Best Regards,
WenYuan Wang
Microsoft Online Community Support