all groups > inetserver iis > november 2005 >
Hi... I have a couple of questions about ISAPI functioning... 1) It seems by default that when the isapi handles the request and returns a SF_STATUS_REQ_FINISHED, whatever response is made is made with the same protocol that the original request was in. I.e. if the incoming request was HTTP/1.1, using the ISAPI api to construct and send a response will respond with HTTP/1.1. Is there any way to change the response protocol on answers generated by the ISAPI filter? I would like to down-grade to HTTP/1.0 on answers that are finished by the ISAPI filter. Currently my filter, based on some characteristics in the original request will pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER, (PVOID) httpCode, (DWORD) addHeader, 0); return SF_STATUS_REQ_FINISHED; Since there isn't any more to the request, I'd rather use HTTP/1.0 to get the server to close the connection and be done with it, rather than respond with HTTP/1.1 which leaves it up to the caller. 2) Is there much (any) difference between ServerSupportFunction (..., SF_REQ_SEND_RESPONSE_HEADER, ...) and the AddResponseHeader() function? Thanks _mark
Hi Mark, 1) The HTTP spec says that both the client and server should report the highest version of the spec that they should support. RFC2616, section 3.1 states: "The HTTP version of an application is the highest HTTP version for which the application is at least conditionally compliant." Because of this, IIS will always return "HTTP/1.1" in the version field of its responses. So, when IIS sends a response to an HTTP/1.0 client, it does follow HTTP/1.0 semantics so these responses will always be compatible with the client. For example, IIS will not send a chunked transfer encoded response to an HTTP/1.0 client because that feature is not supported in 1.0. Note that it is possible for specific ISAPI extensions and filters to introduce problems here. To use the above example, if an ISAPI extension returns chunked transfer encoded responses, it is the responsiblity of that ISAPI to check the client HTTP version and provide a non-chunked response to 1.0 clients. In the case of keep-alive management, IIS will take care of watching the client version and setting an appropriate connection response header as long as you use a function that is intended to send headers (ie. like SF_REQ_SEND_RESPONSE_HEADER.) You should *never* add a connection header to the response unless you are skipping IIS's send header functions and sending the whole response as a raw stream through WriteClient (or HSE_REQ_TRANSMIT_FILE or HSE_REQ_VECTOR_SEND without using the HSE_IO_SEND_HEADERS flag.) In your specific situation, you are sending a response from a filter and trying to ensure that the connection is closed. You are in luck here. All responses sent from a filter will always close the connection because SF_STATUS_REQ_FINISHED will close the connection - and SF_STATUS_REQ_FINISHED_AND_KEEP_CONN was never implemented. (In ISAPI extensions, the rules about when a connection is kept open or closed is somewhat complicated and it would confused things to try and explain it all here...) 2) There is a very substantial difference between SF_REQ_SEND_RESPONSE_HEADER and AddResponseHeaders. The former function sends the response headers immediately, and the filter is then responsible for sending out the whole response and returning SF_STATUS_REQ_FINISHED. The latter function does not send out headers at all. IIS has several internally generated headers (like the Server header, for example) that are inserted into responses automatically. AddResponseHeaders simply adds your headers to that list of internal headers. The headers are only sent out when someone calls one of the functions to send headers. Note that if nobody calls one of the send header functions (ie. by sending the response as a raw stream through WriteClient), then they would never be sent at all. As you develop your filter, I would *strongly* encourage you to look at the traffic between the client and server using a packet sniffer, like Microsoft Network Monitor. Without understanding exactly what is happening on the wire, it is highly likely that you will end up sending out incorrectly formed HTTP data. Note that this assumes that you have a working knowledge of the details of the HTTP protocol, which should be considered a requirement for developing with ISAPI. The HTTP spec can be found in RFC2616 ( http://www.faqs.org/rfcs/rfc2616.html). I hope that this information is helpful, -Wade A. Hilmo, -Microsoft PS: If you have any other questions about this or any other ISAPI issue, please feel free to post them to microsoft.public.platformsdk.internet.server.isapi-dev, which exists for this purpose. [quoted text, click to view] "Mark" <mmodrall@nospam.nospam> wrote in message
news:FBD9E9B9-224A-48EF-8ABC-A66C64391F5E@microsoft.com...
> Hi...
>
> I have a couple of questions about ISAPI functioning...
>
> 1) It seems by default that when the isapi handles the request and returns
a
> SF_STATUS_REQ_FINISHED, whatever response is made is made with the same
> protocol that the original request was in. I.e. if the incoming request
was
> HTTP/1.1, using the ISAPI api to construct and send a response will
respond
> with HTTP/1.1. Is there any way to change the response protocol on
answers
> generated by the ISAPI filter? I would like to down-grade to HTTP/1.0 on
> answers that are finished by the ISAPI filter.
>
> Currently my filter, based on some characteristics in the original request
> will
>
> pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER,
> (PVOID) httpCode, (DWORD) addHeader, 0);
> return SF_STATUS_REQ_FINISHED;
>
> Since there isn't any more to the request, I'd rather use HTTP/1.0 to get
> the server to close the connection and be done with it, rather than
respond
> with HTTP/1.1 which leaves it up to the caller.
>
> 2) Is there much (any) difference between ServerSupportFunction (...,
> SF_REQ_SEND_RESPONSE_HEADER, ...) and the AddResponseHeader() function?
>
> Thanks
> _mark
>
Hi Wade... Thanks for the very thorough response. To be a little more specific, our ISAPI was trying to return a 403 to some bot we detected and we wanted to not spend any more time on it. We had it returning a status 403 with a Connection: Close header, but we found at the switch that this bot still had a fairly large number of connections in "time wait" state. We figured that was because the 1.1 client wasn't letting go of the connection. It was a bit of a vain hope that down-grading to 1.0 might get them to let go. But it sounds from your description like the server should have already closed the connection. Perhaps the several dozen connections in time wait state were the normal time it takes to clean up after, even with a quick iis close. Thanks _mark [quoted text, click to view] "Wade A. Hilmo [MS]" wrote: > Hi Mark, > > 1) The HTTP spec says that both the client and server should report the > highest version of the spec that they should support. RFC2616, section 3.1 > states: > > "The HTTP version of an application is the highest HTTP version for which > the application is at least conditionally compliant." > > Because of this, IIS will always return "HTTP/1.1" in the version field of > its responses. So, when IIS sends a response to an HTTP/1.0 client, it does > follow HTTP/1.0 semantics so these responses will always be compatible with > the client. For example, IIS will not send a chunked transfer encoded > response to an HTTP/1.0 client because that feature is not supported in 1.0. > > Note that it is possible for specific ISAPI extensions and filters to > introduce problems here. To use the above example, if an ISAPI extension > returns chunked transfer encoded responses, it is the responsiblity of that > ISAPI to check the client HTTP version and provide a non-chunked response to > 1.0 clients. > > In the case of keep-alive management, IIS will take care of watching the > client version and setting an appropriate connection response header as long > as you use a function that is intended to send headers (ie. like > SF_REQ_SEND_RESPONSE_HEADER.) You should *never* add a connection header to > the response unless you are skipping IIS's send header functions and sending > the whole response as a raw stream through WriteClient (or > HSE_REQ_TRANSMIT_FILE or HSE_REQ_VECTOR_SEND without using the > HSE_IO_SEND_HEADERS flag.) > > In your specific situation, you are sending a response from a filter and > trying to ensure that the connection is closed. You are in luck here. All > responses sent from a filter will always close the connection because > SF_STATUS_REQ_FINISHED will close the connection - and > SF_STATUS_REQ_FINISHED_AND_KEEP_CONN was never implemented. > > (In ISAPI extensions, the rules about when a connection is kept open or > closed is somewhat complicated and it would confused things to try and > explain it all here...) > > 2) There is a very substantial difference between > SF_REQ_SEND_RESPONSE_HEADER and AddResponseHeaders. The former function > sends the response headers immediately, and the filter is then responsible > for sending out the whole response and returning SF_STATUS_REQ_FINISHED. > The latter function does not send out headers at all. IIS has several > internally generated headers (like the Server header, for example) that are > inserted into responses automatically. AddResponseHeaders simply adds your > headers to that list of internal headers. The headers are only sent out > when someone calls one of the functions to send headers. Note that if > nobody calls one of the send header functions (ie. by sending the response > as a raw stream through WriteClient), then they would never be sent at all. > > As you develop your filter, I would *strongly* encourage you to look at the > traffic between the client and server using a packet sniffer, like Microsoft > Network Monitor. Without understanding exactly what is happening on the > wire, it is highly likely that you will end up sending out incorrectly > formed HTTP data. Note that this assumes that you have a working knowledge > of the details of the HTTP protocol, which should be considered a > requirement for developing with ISAPI. The HTTP spec can be found in > RFC2616 ( http://www.faqs.org/rfcs/rfc2616.html). > > I hope that this information is helpful, > -Wade A. Hilmo, > -Microsoft > > PS: If you have any other questions about this or any other ISAPI issue, > please feel free to post them to > microsoft.public.platformsdk.internet.server.isapi-dev, which exists for > this purpose. > > "Mark" <mmodrall@nospam.nospam> wrote in message > news:FBD9E9B9-224A-48EF-8ABC-A66C64391F5E@microsoft.com... > > Hi... > > > > I have a couple of questions about ISAPI functioning... > > > > 1) It seems by default that when the isapi handles the request and returns > a > > SF_STATUS_REQ_FINISHED, whatever response is made is made with the same > > protocol that the original request was in. I.e. if the incoming request > was > > HTTP/1.1, using the ISAPI api to construct and send a response will > respond > > with HTTP/1.1. Is there any way to change the response protocol on > answers > > generated by the ISAPI filter? I would like to down-grade to HTTP/1.0 on > > answers that are finished by the ISAPI filter. > > > > Currently my filter, based on some characteristics in the original request > > will > > > > pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER, > > (PVOID) httpCode, (DWORD) addHeader, 0); > > return SF_STATUS_REQ_FINISHED; > > > > Since there isn't any more to the request, I'd rather use HTTP/1.0 to get > > the server to close the connection and be done with it, rather than > respond > > with HTTP/1.1 which leaves it up to the caller. > > > > 2) Is there much (any) difference between ServerSupportFunction (..., > > SF_REQ_SEND_RESPONSE_HEADER, ...) and the AddResponseHeader() function? > > > > Thanks > > _mark > > > >
Hi Mark, When connections are closed on the server side, it is normal for the connections to go into TIME_WAIT. This should not be a problem. In my opinion, best way for your filter to drop these connections is to identify them on SF_NOTIFY_PREPROC_HEADERS and then do a SetLastError to ERROR_FILE_NOT_FOUND and return SF_STATUS_REQ_ERROR. If you do this, IIS will send an asynchronous 404 to the client. This is more efficient than doing synchronous sends, which is what your current code is doing. Alternately, you could send a 500 response back by setting a different error in SetLastError. Thank you, -Wade A. Hilmo, -Microsoft [quoted text, click to view] "Mark" <mmodrall@nospam.nospam> wrote in message news:C473E06B-EDCD-4218-A760-2FC3B19A4A5C@microsoft.com... > Hi Wade... > > Thanks for the very thorough response. To be a little more specific, our > ISAPI was trying to return a 403 to some bot we detected and we wanted to not > spend any more time on it. We had it returning a status 403 with a > Connection: Close header, but we found at the switch that this bot still had > a fairly large number of connections in "time wait" state. We figured that > was because the 1.1 client wasn't letting go of the connection. > > It was a bit of a vain hope that down-grading to 1.0 might get them to let > go. But it sounds from your description like the server should have already > closed the connection. Perhaps the several dozen connections in time wait > state were the normal time it takes to clean up after, even with a quick iis > close. > > Thanks > _mark > > > "Wade A. Hilmo [MS]" wrote: > > > Hi Mark, > > > > 1) The HTTP spec says that both the client and server should report the > > highest version of the spec that they should support. RFC2616, section 3.1 > > states: > > > > "The HTTP version of an application is the highest HTTP version for which > > the application is at least conditionally compliant." > > > > Because of this, IIS will always return "HTTP/1.1" in the version field of > > its responses. So, when IIS sends a response to an HTTP/1.0 client, it does > > follow HTTP/1.0 semantics so these responses will always be compatible with > > the client. For example, IIS will not send a chunked transfer encoded > > response to an HTTP/1.0 client because that feature is not supported in 1.0. > > > > Note that it is possible for specific ISAPI extensions and filters to > > introduce problems here. To use the above example, if an ISAPI extension > > returns chunked transfer encoded responses, it is the responsiblity of that > > ISAPI to check the client HTTP version and provide a non-chunked response to > > 1.0 clients. > > > > In the case of keep-alive management, IIS will take care of watching the > > client version and setting an appropriate connection response header as long > > as you use a function that is intended to send headers (ie. like > > SF_REQ_SEND_RESPONSE_HEADER.) You should *never* add a connection header to > > the response unless you are skipping IIS's send header functions and sending > > the whole response as a raw stream through WriteClient (or > > HSE_REQ_TRANSMIT_FILE or HSE_REQ_VECTOR_SEND without using the > > HSE_IO_SEND_HEADERS flag.) > > > > In your specific situation, you are sending a response from a filter and > > trying to ensure that the connection is closed. You are in luck here. All > > responses sent from a filter will always close the connection because > > SF_STATUS_REQ_FINISHED will close the connection - and > > SF_STATUS_REQ_FINISHED_AND_KEEP_CONN was never implemented. > > > > (In ISAPI extensions, the rules about when a connection is kept open or > > closed is somewhat complicated and it would confused things to try and > > explain it all here...) > > > > 2) There is a very substantial difference between > > SF_REQ_SEND_RESPONSE_HEADER and AddResponseHeaders. The former function > > sends the response headers immediately, and the filter is then responsible > > for sending out the whole response and returning SF_STATUS_REQ_FINISHED. > > The latter function does not send out headers at all. IIS has several > > internally generated headers (like the Server header, for example) that are > > inserted into responses automatically. AddResponseHeaders simply adds your > > headers to that list of internal headers. The headers are only sent out > > when someone calls one of the functions to send headers. Note that if > > nobody calls one of the send header functions (ie. by sending the response > > as a raw stream through WriteClient), then they would never be sent at all. > > > > As you develop your filter, I would *strongly* encourage you to look at the > > traffic between the client and server using a packet sniffer, like Microsoft > > Network Monitor. Without understanding exactly what is happening on the > > wire, it is highly likely that you will end up sending out incorrectly > > formed HTTP data. Note that this assumes that you have a working knowledge > > of the details of the HTTP protocol, which should be considered a > > requirement for developing with ISAPI. The HTTP spec can be found in > > RFC2616 ( http://www.faqs.org/rfcs/rfc2616.html). > > > > I hope that this information is helpful, > > -Wade A. Hilmo, > > -Microsoft > > > > PS: If you have any other questions about this or any other ISAPI issue, > > please feel free to post them to > > microsoft.public.platformsdk.internet.server.isapi-dev, which exists for > > this purpose. > > > > "Mark" <mmodrall@nospam.nospam> wrote in message > > news:FBD9E9B9-224A-48EF-8ABC-A66C64391F5E@microsoft.com... > > > Hi... > > > > > > I have a couple of questions about ISAPI functioning... > > > > > > 1) It seems by default that when the isapi handles the request and returns > > a > > > SF_STATUS_REQ_FINISHED, whatever response is made is made with the same > > > protocol that the original request was in. I.e. if the incoming request > > was > > > HTTP/1.1, using the ISAPI api to construct and send a response will > > respond > > > with HTTP/1.1. Is there any way to change the response protocol on > > answers > > > generated by the ISAPI filter? I would like to down-grade to HTTP/1.0 on > > > answers that are finished by the ISAPI filter. > > > > > > Currently my filter, based on some characteristics in the original request > > > will > > > > > > pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER, > > > (PVOID) httpCode, (DWORD) addHeader, 0); > > > return SF_STATUS_REQ_FINISHED; > > > > > > Since there isn't any more to the request, I'd rather use HTTP/1.0 to get > > > the server to close the connection and be done with it, rather than > > respond > > > with HTTP/1.1 which leaves it up to the caller. > > > > > > 2) Is there much (any) difference between ServerSupportFunction (...,
Hi Wade... Thanks for more good suggestions. They do raise a few more questions though; I hope I'm not wearing your patience thin. First, the bot in question was hitting us with a denial-of-service attack (intentionally or unintentionally), and even though we configured our ISAPI to handle/reject the requests from that specific ip, we were still seeing lots of connections in the TIME_WAIT state (from the rejections) and a lot of queuing in IIS. Our experience has been that once IIS/ASP gets past about 50-75 requests queued, it's in a death spiral and crashes. The thought was that the bot sockets in the time wait state was chewing resources and causing the backlog. When they started blocking the bot ip at the switch (a more manually intensive thing to maintain), the servers righted and we got service back. So if you're getting hammered by pernicious traffic, would it still be okay to have all these time wait connections? Secondly, when you return the SF_REQ_STATUS_ERROR status and let the server finish the request asynchronously, a) can you still do any setting of response headers, or will that be superceded by the error processing? b) would possibly deferred closing perhaps exacerbate the connection/load problem? Thanks _mark [quoted text, click to view] "Wade A. Hilmo [MS]" wrote:
> Hi Mark,
>
> When connections are closed on the server side, it is normal for the
> connections to go into TIME_WAIT. This should not be a problem.
>
> In my opinion, best way for your filter to drop these connections is to
> identify them on SF_NOTIFY_PREPROC_HEADERS and then do a SetLastError to
> ERROR_FILE_NOT_FOUND and return SF_STATUS_REQ_ERROR. If you do this, IIS
> will send an asynchronous 404 to the client. This is more efficient than
> doing synchronous sends, which is what your current code is doing.
> Alternately, you could send a 500 response back by setting a different error
> in SetLastError.
>
> Thank you,
> -Wade A. Hilmo,
> -Microsoft
Hi Mark, IIS itself does not queue requests. If you dump the requests in SF_NOTIFY_PREPROC_HEADERS, then these particular requests should not get into ASP and should have no direct effect on queuing. IIS does not do a whole lot of processing before PREPROC_HEADERS, so it's about the best place to do this. Of course, dumping them at your router will be far more efficient than anything you can do inside of IIS. Just as another note, I've seen our UrlScan tool (which can dump requests using SF_STATUS_REQ_ERROR) have a large positive impact on machines that are receiving a high volume of unwanted requests. I would imagine that your filter can do as well. As far as TIME_WAIT, there is no way to avoid it other than to do a keep-alive response, which is not what you want to do here. For what it's worth, I've seen thousands of sockets in TIME_WAIT without a serious adverse effect. Finally, I am not positive, but I would suspect that SF_STATUS_REQ_ERROR will prevent any custom headers from going out. The easiest way to know for sure is to use AddResponseHeaders before returning and then take a look at the response. If you find that my guess is wrong and the headers are going out, it would be good to post that back to this thread so that others reading it get the correct information. Thank you, -Wade A. Hilmo, -Microsoft [quoted text, click to view] "Mark" <mmodrall@nospam.nospam> wrote in message
news:683270FC-94DE-4246-9B6A-4ED0C61A1F37@microsoft.com...
> Hi Wade...
>
> Thanks for more good suggestions. They do raise a few more questions
> though; I hope I'm not wearing your patience thin.
>
> First, the bot in question was hitting us with a denial-of-service attack
> (intentionally or unintentionally), and even though we configured our
ISAPI
> to handle/reject the requests from that specific ip, we were still seeing
> lots of connections in the TIME_WAIT state (from the rejections) and a lot
of
> queuing in IIS. Our experience has been that once IIS/ASP gets past about
> 50-75 requests queued, it's in a death spiral and crashes. The thought
was
> that the bot sockets in the time wait state was chewing resources and
causing
> the backlog. When they started blocking the bot ip at the switch (a more
> manually intensive thing to maintain), the servers righted and we got
service
> back. So if you're getting hammered by pernicious traffic, would it still
be
> okay to have all these time wait connections?
>
> Secondly, when you return the SF_REQ_STATUS_ERROR status and let the
server
> finish the request asynchronously,
> a) can you still do any setting of response headers, or will that be
> superceded by the error processing?
>
> b) would possibly deferred closing perhaps exacerbate the connection/load
> problem?
>
> Thanks
> _mark
>
>
> "Wade A. Hilmo [MS]" wrote:
>
> > Hi Mark,
> >
> > When connections are closed on the server side, it is normal for the
> > connections to go into TIME_WAIT. This should not be a problem.
> >
> > In my opinion, best way for your filter to drop these connections is to
> > identify them on SF_NOTIFY_PREPROC_HEADERS and then do a SetLastError to
> > ERROR_FILE_NOT_FOUND and return SF_STATUS_REQ_ERROR. If you do this,
IIS
> > will send an asynchronous 404 to the client. This is more efficient
than
> > doing synchronous sends, which is what your current code is doing.
> > Alternately, you could send a 500 response back by setting a different
error
> > in SetLastError.
> >
> > Thank you,
> > -Wade A. Hilmo,
> > -Microsoft
>
Hi Wade...
Our ISAPI uses
pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER,
(PVOID) httpCode, (DWORD) addHeader, 0);
to set the custom headers instead of AddResponseHeaders(), but I did a quick
test just changing the isapi return code to SF_STATUS_REQ_ERROR. Then I ran
it up the flag pole with requests designed to trigger the ISAPI response.
The interesting part is that I got my custom headers *and* a canned IIS
response; in effect I got two responses instead of one. Since I didn't add a
SetLastError() call, it just took whatever was at hand from other operation.
Below is the output:
HTTP/1.1 403 Request Denied
Server: Microsoft-IIS/5.0
Date: Wed, 30 Nov 2005 23:09:09 GMT
Content-Length: 80
Connection: Close
<html><head><META HTTP-EQUIV="Refresh" CONTENT="0;URL=403.html"></head></html>
HTTP/1.1 500 Server Error
Server: Microsoft-IIS/5.0
Date: Wed, 30 Nov 2005 23:09:09 GMT
Content-Type: text/html
Content-Length: 111
<html><head><title>Error</title></head><body>The data area passed to a
system call is too small. </body></html>
Curious...
After thinking about some of your responses, it did seem like the TIME_WAIT
wouldn't be related to the asp queuing problem. The only thing I can think
of (and it's somewhat round-about) is that we're getting *so many* requests
from this bot that even our ISAPI filter is having trouble keeping up. The
cpu usage gets driven through the roof processing the filter requests causing
slowdown in getting to the asp engine for the legitimate requests coming in,
causing the queuing.
Thanks
Mark
If you return SF_STATUS_REQ_ERROR in SF_NOTIFY_PREPROC_HEADERS, you should NOT call SF_REQ_SEND_RESPONSE_HEADER or else you *will* get multiple response headers. This is because when you call SF_REQ_SEND_RESPONSE_HEADER, IIS writes something to the output stream. Then, when you return SF_STATUS_REQ_ERROR in SF_NOTIFY_PREPROC_HEADERS, IIS detects that and starts to send the pre-canned responses... which has its own headers/entity body. Here is sample Filter code that rejects based on a request characteristic and talks more about how SF_STATUS_REQ_ERROR in SF_NOTIFY_PREPROC_HEADERS work: http://blogs.msdn.com/david.wang/archive/2005/07/01/HOWTO_ISAPI_Filter_rejecting_requests_from_SF_NOTIFY_PREPROC_HEADERS_based_on_HTTP_Referer.aspx Personally, I would not bother with the filter until you have identified the resource bottleneck that is under DoS. You have to realize that any server can be DoS'd purely by legitimate requests -- the attacker simply has to consume some resource required for your server to remain publicly accessible, whether it is bandwidth between the server and Internet, CPU cycles, Memory (for TCP connections), or some exploitable logic bug. The challenge is for you to identify *which* of the resources is being consumed and deflect that. You are mentioning ASP queuing -- so maybe the DoS is targeting an ASP page? Or the CPU is so loaded doing something else (maybe you have some other ISAPI Extension/Filter that is doing a lot of work) that ASP does not have a chance to execute? Deflecting at the router level is the fastest way to make progress, but you also want to know what is vulnerable to DoS on your server and fix that (most people have tons of code vulnerable to DoS on their webservers, intentionally, to do useful work). -- //David IIS http://blogs.msdn.com/David.Wang This posting is provided "AS IS" with no warranties, and confers no rights. // [quoted text, click to view] "Mark" <mmodrall@nospam.nospam> wrote in message
news:12F8BAB0-5391-46B8-ACAB-8BCC329B3A3A@microsoft.com...
Hi Wade... Our ISAPI uses pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER, (PVOID) httpCode, (DWORD) addHeader, 0); to set the custom headers instead of AddResponseHeaders(), but I did a quick test just changing the isapi return code to SF_STATUS_REQ_ERROR. Then I ran it up the flag pole with requests designed to trigger the ISAPI response. The interesting part is that I got my custom headers *and* a canned IIS response; in effect I got two responses instead of one. Since I didn't add a SetLastError() call, it just took whatever was at hand from other operation. Below is the output: HTTP/1.1 403 Request Denied Server: Microsoft-IIS/5.0 Date: Wed, 30 Nov 2005 23:09:09 GMT Content-Length: 80 Connection: Close <html><head><META HTTP-EQUIV="Refresh" CONTENT="0;URL=403.html"></head></html> HTTP/1.1 500 Server Error Server: Microsoft-IIS/5.0 Date: Wed, 30 Nov 2005 23:09:09 GMT Content-Type: text/html Content-Length: 111 <html><head><title>Error</title></head><body>The data area passed to a system call is too small. </body></html> Curious... After thinking about some of your responses, it did seem like the TIME_WAIT wouldn't be related to the asp queuing problem. The only thing I can think of (and it's somewhat round-about) is that we're getting *so many* requests from this bot that even our ISAPI filter is having trouble keeping up. The cpu usage gets driven through the roof processing the filter requests causing slowdown in getting to the asp engine for the legitimate requests coming in, causing the queuing. Thanks Mark
Hi Mark, What you are seeing is expected. As I mentioned earlier, SF_REQ_SEND_RESPONSE_HEADER sends a set of response headers immediately. The reason you see two is that you see yours, and then you see the one that IIS sends because of the error return. As a rule, you *must* return SF_STATUS_REQ_FINISHED any time that you use SF_REQ_SEND_RESPONSE_HEADER or you will see this. You should use AddResponseHeaders instead for this test. I hope that this makes sense, -Wade A. Hilmo, -Microsoft [quoted text, click to view] "Mark" <mmodrall@nospam.nospam> wrote in message
news:12F8BAB0-5391-46B8-ACAB-8BCC329B3A3A@microsoft.com...
> Hi Wade...
>
> Our ISAPI uses
> pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER,
> (PVOID) httpCode, (DWORD) addHeader, 0);
>
> to set the custom headers instead of AddResponseHeaders(), but I did a
quick
> test just changing the isapi return code to SF_STATUS_REQ_ERROR. Then I
ran
> it up the flag pole with requests designed to trigger the ISAPI response.
> The interesting part is that I got my custom headers *and* a canned IIS
> response; in effect I got two responses instead of one. Since I didn't
add a
> SetLastError() call, it just took whatever was at hand from other
operation.
> Below is the output:
>
>
> HTTP/1.1 403 Request Denied
> Server: Microsoft-IIS/5.0
> Date: Wed, 30 Nov 2005 23:09:09 GMT
> Content-Length: 80
> Connection: Close
>
> <html><head><META HTTP-EQUIV="Refresh"
CONTENT="0;URL=403.html"></head></html>
>
>
> HTTP/1.1 500 Server Error
> Server: Microsoft-IIS/5.0
> Date: Wed, 30 Nov 2005 23:09:09 GMT
> Content-Type: text/html
> Content-Length: 111
>
> <html><head><title>Error</title></head><body>The data area passed to a
> system call is too small. </body></html>
>
>
> Curious...
>
> After thinking about some of your responses, it did seem like the
TIME_WAIT
> wouldn't be related to the asp queuing problem. The only thing I can
think
> of (and it's somewhat round-about) is that we're getting *so many*
requests
> from this bot that even our ISAPI filter is having trouble keeping up.
The
> cpu usage gets driven through the roof processing the filter requests
causing
> slowdown in getting to the asp engine for the legitimate requests coming
in,
> causing the queuing.
>
> Thanks
> Mark
>
Hi David... Thanks for responding. I take your point about identifying the bottleneck. Unfortunately, I suspect now that there might be some involvement with the filter in this case. What ultimately crashes IIS is the ASP queuing. Over the years, we've seen that once asp queuing gets past a certain level, IIS gets unstable and can't recover even if the load on the system is ameliorated. The question is why/how are so many requests ending up queued? It's unclear whether the intent of the attack was DoS or just a particularly aggressive bot. Our site is all asp pages, so it's not an attempt to specifically target a soft spot. The unwanted traffic is definitely from a bot, though. It's coming from a particular ip and the user agent string is Mozilla/5.0+(compatible;+Googlebot/2.1;++ http://www.google.com/bot.html), though I doubt that's a legit Google ua string. It was hitting us with between 4-10 qps per server. When the traffic was allowed to get to the iis servers, the symptoms noted where extremely high cpu load followed by asp queuing and ultimately iis death. The interesting thing was that once the ip was identified, we added it to our isapi filter's config and kept all that traffic from hitting the asp engine - but while the time to death was delayed, the rest of the symptoms stayed the same. Our isapi checks a number of things, based on a config file, including user agent strings, client ips, query strings, cookies, etc, so my working hypothesis on the cause of death of IIS was that the volume of traffic was high enough that even taking the time in the isapi to run through the list of fields to check caused a spike in the cpu. The spike in the cpu slowed down the processing of legit requests causing asp queuing, asp queuing got to the IIS tipping point and the cow went over. Only when we blocked it at the switch did things get better. Anyway, that's my theory; it seems a bit of a stretch but it's the closest I can get with the observed conditions. Thanks _mark [quoted text, click to view] "David Wang [Msft]" wrote:
> Personally, I would not bother with the filter until you have identified the
> resource bottleneck that is under DoS.
>
> You have to realize that any server can be DoS'd purely by legitimate
> requests -- the attacker simply has to consume some resource required for
> your server to remain publicly accessible, whether it is bandwidth between
> the server and Internet, CPU cycles, Memory (for TCP connections), or some
> exploitable logic bug. The challenge is for you to identify *which* of the
> resources is being consumed and deflect that.
>
> You are mentioning ASP queuing -- so maybe the DoS is targeting an ASP page?
> Or the CPU is so loaded doing something else (maybe you have some other
> ISAPI Extension/Filter that is doing a lot of work) that ASP does not have a
> chance to execute?
>
> Deflecting at the router level is the fastest way to make progress, but you
> also want to know what is vulnerable to DoS on your server and fix that
> (most people have tons of code vulnerable to DoS on their webservers,
> intentionally, to do useful work).
Hi Wade... Thanks again for all your help. I substituted AddResponseHeaders for the send function and while the response changed slightly, the AddResponseHeaders were still included in the output. Here's what comes out: HTTP/1.1 500 Server Error Server: Microsoft-IIS/5.0 Date: Thu, 01 Dec 2005 15:10:28 GMT Connection: close Content-Length: 80 Connection: Close <html><head><META HTTP-EQUIV="Refresh" CONTENT="0;URL=403.html"></head></html> Content-Type: text/html Content-Length: 111 <html><head><title>Error</title></head><body>The data area passed to a system call is too small. </body></html> Thanks _mark [quoted text, click to view] "Wade A. Hilmo [MS]" wrote:
> Hi Mark,
>
> What you are seeing is expected. As I mentioned earlier,
> SF_REQ_SEND_RESPONSE_HEADER sends a set of response headers immediately.
> The reason you see two is that you see yours, and then you see the one that
> IIS sends because of the error return. As a rule, you *must* return
> SF_STATUS_REQ_FINISHED any time that you use SF_REQ_SEND_RESPONSE_HEADER or
> you will see this.
>
> You should use AddResponseHeaders instead for this test.
>
> I hope that this makes sense,
> -Wade A. Hilmo,
> -Microsoft
>
> "Mark" <mmodrall@nospam.nospam> wrote in message
> news:12F8BAB0-5391-46B8-ACAB-8BCC329B3A3A@microsoft.com...
> > Hi Wade...
> >
> > Our ISAPI uses
> > pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER,
> > (PVOID) httpCode, (DWORD) addHeader, 0);
> >
> > to set the custom headers instead of AddResponseHeaders(), but I did a
> quick
> > test just changing the isapi return code to SF_STATUS_REQ_ERROR. Then I
> ran
> > it up the flag pole with requests designed to trigger the ISAPI response.
> > The interesting part is that I got my custom headers *and* a canned IIS
> > response; in effect I got two responses instead of one. Since I didn't
> add a
> > SetLastError() call, it just took whatever was at hand from other
> operation.
> > Below is the output:
> >
> >
> > HTTP/1.1 403 Request Denied
> > Server: Microsoft-IIS/5.0
> > Date: Wed, 30 Nov 2005 23:09:09 GMT
> > Content-Length: 80
> > Connection: Close
> >
> > <html><head><META HTTP-EQUIV="Refresh"
> CONTENT="0;URL=403.html"></head></html>
> >
> >
> > HTTP/1.1 500 Server Error
> > Server: Microsoft-IIS/5.0
> > Date: Wed, 30 Nov 2005 23:09:09 GMT
> > Content-Type: text/html
> > Content-Length: 111
> >
> > <html><head><title>Error</title></head><body>The data area passed to a
> > system call is too small. </body></html>
> >
> >
> > Curious...
> >
> > After thinking about some of your responses, it did seem like the
> TIME_WAIT
> > wouldn't be related to the asp queuing problem. The only thing I can
> think
> > of (and it's somewhat round-about) is that we're getting *so many*
> requests
> > from this bot that even our ISAPI filter is having trouble keeping up.
> The
> > cpu usage gets driven through the roof processing the filter requests
> causing
> > slowdown in getting to the asp engine for the legitimate requests coming
> in,
> > causing the queuing.
> >
> > Thanks
> > Mark
> >
>
>
It looks like you used AddResponseHeaders to add both headers and entity body (the following part seems to be sent by your ISAPI Filter): Content-Length: 80\r\n Connection: Close\r\n \r\n <html><head><META HTTP-EQUIV="Refresh" CONTENT="0;URL=403.html"></head></html> And then this is what IIS was sending on the 500 Response because you return SF_REQ_STATUS_ERROR: Content-Type: text/html\r\n Content-Length: 111\r\n \r\n <html><head><title>Error</title></head><body>The data area passed to a system call is too small. </body></html> I'm not certain what exactly you are trying to accomplish. You are doing fast-path rejection by returning SF_REQ_STATUS_ERROR, so why the custom entity body? It seems odd that when you reject a request, you are asking the client to make yet another request for a 403.html from the server. You basically double the number of requests to your server for every rejection... -- //David IIS http://blogs.msdn.com/David.Wang This posting is provided "AS IS" with no warranties, and confers no rights. // [quoted text, click to view] "Mark" <mmodrall@nospam.nospam> wrote in message
news:E4B9B84E-E27E-46A7-AE62-7C59A1D41B23@microsoft.com...
Hi Wade... Thanks again for all your help. I substituted AddResponseHeaders for the send function and while the response changed slightly, the AddResponseHeaders were still included in the output. Here's what comes out: HTTP/1.1 500 Server Error Server: Microsoft-IIS/5.0 Date: Thu, 01 Dec 2005 15:10:28 GMT Connection: close Content-Length: 80 Connection: Close <html><head><META HTTP-EQUIV="Refresh" CONTENT="0;URL=403.html"></head></html> Content-Type: text/html Content-Length: 111 <html><head><title>Error</title></head><body>The data area passed to a system call is too small. </body></html> Thanks _mark [quoted text, click to view] "Wade A. Hilmo [MS]" wrote:
> Hi Mark,
>
> What you are seeing is expected. As I mentioned earlier,
> SF_REQ_SEND_RESPONSE_HEADER sends a set of response headers immediately.
> The reason you see two is that you see yours, and then you see the one
that
> IIS sends because of the error return. As a rule, you *must* return
> SF_STATUS_REQ_FINISHED any time that you use SF_REQ_SEND_RESPONSE_HEADER
or
> you will see this.
>
> You should use AddResponseHeaders instead for this test.
>
> I hope that this makes sense,
> -Wade A. Hilmo,
> -Microsoft
>
> "Mark" <mmodrall@nospam.nospam> wrote in message
> news:12F8BAB0-5391-46B8-ACAB-8BCC329B3A3A@microsoft.com...
> > Hi Wade...
> >
> > Our ISAPI uses
> > pfc->ServerSupportFunction(pfc, SF_REQ_SEND_RESPONSE_HEADER,
> > (PVOID) httpCode, (DWORD) addHeader, 0);
> >
> > to set the custom headers instead of AddResponseHeaders(), but I did a
> quick
> > test just changing the isapi return code to SF_STATUS_REQ_ERROR. Then I
> ran
> > it up the flag pole with requests designed to trigger the ISAPI
response.
> > The interesting part is that I got my custom headers *and* a canned IIS
> > response; in effect I got two responses instead of one. Since I didn't
> add a
> > SetLastError() call, it just took whatever was at hand from other
> operation.
> > Below is the output:
> >
> >
> > HTTP/1.1 403 Request Denied
> > Server: Microsoft-IIS/5.0
> > Date: Wed, 30 Nov 2005 23:09:09 GMT
> > Content-Length: 80
> > Connection: Close
> >
> > <html><head><META HTTP-EQUIV="Refresh"
> CONTENT="0;URL=403.html"></head></html>
> >
> >
> > HTTP/1.1 500 Server Error
> > Server: Microsoft-IIS/5.0
> > Date: Wed, 30 Nov 2005 23:09:09 GMT
> > Content-Type: text/html
> > Content-Length: 111
> >
> > <html><head><title>Error</title></head><body>The data area passed to a
> > system call is too small. </body></html>
> >
> >
> > Curious...
> >
> > After thinking about some of your responses, it did seem like the
> TIME_WAIT
> > wouldn't be related to the asp queuing problem. The only thing I can
> think
> > of (and it's somewhat round-about) is that we're getting *so many*
> requests
> > from this bot that even our ISAPI filter is having trouble keeping up.
> The
> > cpu usage gets driven through the roof processing the filter requests
> causing
> > slowdown in getting to the asp engine for the legitimate requests coming
> in,
> > causing the queuing.
> >
> > Thanks
> > Mark
> >
>
>
>
Hi David...
We got here from a number of different paths. Currently our filter does
craft a custom response, uses SEND_RESPONSE_HEADERS and
SF_REQ_STATUS_FINISHED. But when this bot was hitting us, IIS was still
going belly up and dying. The outer symptoms were high cpu load and asp
queuing to the point where IIS gets unstable and dying. We noticed that the
ip from which the bot was hitting us had a large number of sockets in
TIME_WAIT state, and we were wondering if the machine was getting jammed up
with zombie sockets.
Wade suggested using SF_REQ_STATUS_ERROR instead of SF_REQ_STATUS_FINISHED
because he said it was more efficient and that IIS could resolve the request
asynchronously. That prompted me to ask what would happen with the custom
headers (if any) we put in; Wade asked me to give it a try and see, which is
what I was posting in reply.
As for what we had in our custom content, we spit out a little html with the
error status. The html has a meta-refresh instead of a Location: header
because our feeling (rightly or wrongly) is that very few bots look at the
html that closely and follow meta refreshes. The sentiment was that if we
guessed wrong about rejecting someone, the "real" browsers/users would follow
it to a nice explanation page while the bots would drop. Anecdotally
skimming our stats, it looks like
a) the vast majority of bots *don't* follow html meta refreshes and
b) our filter is correct the vast majority of the time, since actual fetches
of the 403.html page is something like 1 per day over the whole farm.
Since we had IIS dying even when our ISAPI filter was configured to reject
the bot, my guess was that we had to tighten the code of our filter, making
it less expensive to do the evaluation and rearrange the config to move those
things most likely to match to the top. Neither of those things are bad
ideas anyway, though I don't know if that will ultimately solve the problem.
Thanks
_mark
Hi Mark, Thanks for your posting! In my opinion, Wade's suggestion that use the SEND_RESPONSE_HEADERS and SF_REQ_STATUS_FINISHED is suitable for the current issue. [quoted text, click to view] >"The outer symptoms were high CPU load and asp queuing to the point where
IIS gets unstable and dying." From the description, I think you caught the IIS hang issue. I suggest you use the Visual Studio IDE to attach into the inetinfo.exe process (in IIS 5.0) or w3wp.exe (in IIS 6.0) to debug the current program and find out the reason. [quoted text, click to view] >"The html has a meta-refresh instead of a Location:"
It seems the meta-refresh is not a good solution under current scenario. Because when the client refresh, it will send another request to IIS. If you have any concern, please let me know. Regards, Yuan Ren [MSFT] Microsoft Online Support
Don't see what you're looking for? Try a search.
|
|
|
