GET
|
Overview
In the previous two chapters we examined how to access the network protocols directly through the Socket class, which allows for precise control over what is sent and received on a socket. While the Socket class is powerful, it is also cumbersome to implement higher-level protocols, such as Hypertext Transfer Protocol (HTTP), because the application has to handle the network transactions as well as build and parse the HTTP commands, which can be quite complex. This chapter introduces the HTTP-related classes, an application-level protocol built on top of TCP for accessing Web resources. The .NET Framework Web classes encapsulate the TCP details for retrieving Web resources into simple and easy- to-use classes. This chapter is divided into several parts. First, we’ll introduce the basics of accessing HTTP-based Web resources via the HttpWebRequest and HttpWebResponse classes. This will be followed by the WebRequest and WebResponse classes that expose a uniform method of accessing any network resource based on request-response, such as files and HTML that can be extended to support other application-level protocols, including FTP (FtpWebRequest and FtpWebResponse) and file sharing. Following that, we’ll cover more advanced topics such as the asynchronous model for accessing Web resources, as well as connection management, establishing secure connections over a Secure Sockets Layer (SSL), and application models that use Web-based classes from different areas (ASP.NET, console applications, etc.). We’ll finish the discussion with code-access security considerations when using the Web classes.
Web Class Basics
There are four classes of interest for accessing Web resources: HttpWebRequest, HttpWebResponse, WebRequest, and WebResponse. The first two are used specifically for HTTP communication, while the second two offer an extensible interface to a variety of application protocols, including HTTP. The Web classes in this chapter are all request-response transactions. The client makes a request to a server to perform some action. The server receives the request and generates a response that is returned to the client. Following the response, the requested entity or content is transmitted. The HTTP protocol defines several actions a client can request of the server. These actions are listed in Table 10-1. The most common method, GET, requests to retrieve a URI resource. For example if a GET request is made for http://www.microsoft.com/ an HTTP request is generated and sent to www.microsoft.com. The server then examines the request, generates a response indicating success, and sends the entity, which is the default HTML page for the Web site. The second column of the table indicates which version of the HTTP protocol introduced the method. |
Table 10-1: HTTP Request Methods
|
||
Method |
Minimum Version |
Description |
GET |
0.9 |
Retrieves a URI resource. |
HEAD |
1 |
Retrieves only the metadata of a URI resource. |
POST |
1 |
Submits a form or other data (such as a SOAP message) |
PUT |
1.1 |
Uploads a file. |
DELETE |
1.1 |
Deletes a URI resource. |
TRACE |
1.1 |
Traces proxy chains. |
OPTIONS |
1.1 |
Queries HTTP server options. |
Additionally, the HTTP protocol defines a series of header options that affect the behavior of the response. These options are simple string name/value pairs. The HTTP protocol defines numerous header options. The majority of these HTTP headers are exposed as properties of the HttpWebRequest, HttpWebResponse, WebRequest, and WebResponse classes. The ContentLength property that maps to the "Content-Length:" header is an example. Notice that some headers are valid only for requests, others for response, some for both request and response, and others for entities only. Table 10-2 lists the HTTP headers exposed as properties.
In addition to many of the headers being exposed as properties, the property Headers returns a WebHeaderCollection object that contains the entire set of headers associated with the request or response. The Add() method of the WebHeaderCollection class can be used to set additional HTTP headers in the collection. Notice that the headers exposed directly as properties cannot be set using the WebHeaderCollection. Instead, they should be accessed through their strongly typed properties.
Table 10-2: HTTP Header Properties
|
|||
Property |
HTTP Header |
Valid Method |
Description |
Accept |
“Accept:” |
Request |
Specifies acceptable media types (e.g., text/html, image/jpeg, etc.) |
Connection |
“Connection:” |
Request, Response |
Controls how connections are handled (e.g., disable persistent connections) |
ContentLength |
“Content-length:” |
Entity |
Specifies the length of the returned entity in bytes |
ContentType |
“Content-type:” |
Entity |
Specifies the media type of the returned entity (e.g., text\html, image\jpeg, etc.) |
Expect |
“Expect:” |
Request |
Indicates expected behavior of client by the server |
IfModifiedSince |
“If-Modified-Since:” |
Request |
Used to verify cache is up-to- date |
Referer |
“Referer:” |
Request |
Specifies the URL of the document containing the reference to the requested URL |
TransferEncoding |
“Transfer-encoding:” |
Request, Response |
Specifies any transformations applied to content (e.g., chunked) |
UserAgent |
“User-Agent:” |
Request |
Specifies the client software issuing the request |
Only the most relevant headers are covered here; for a full listing of the HTTP headers and their corresponding properties, consult the .NET Framework SDK and RFC 2616.
The HttpWebRequest and HttpWebResponse provide HTTP-specific implementations of the WebRequest and WebResponse classes (which are discussed later) and allow access to URI-based HTTP resources. These classes offer a complete implementation of the HTTP 1 and 1.1 protocols.
As we mentioned earlier, HTTP traffic is based on request-response, and the first step of accessing HTTP-based Web resources is to create an instance of an HttpWebRequest object that indicates a Web resource. This is done by calling the Create() method of the WebRequest class as in the following sample:
HttpWebRequest httpRequest;
httpRequest = (HttpWebRequest) WebRequest.Create("http://www.winisp.net/goodrich/default.htm");
Dim httpRequest As HttpWebRequest
httpRequest = WebRequest.Create("http://www.winisp.net/goodrich/default.htm")
The example creates an HttpWebRequest object that points to the URI http://www.winisp.net/goodrich/default.htm. Notice that the constructor for the HttpWebRequest class should not be called - only the WebRequest.Create() method. See the WebRequest and WebResponse section of this chapter for more details on the requirements of calling the WebRequest.Create() method. Once an instance of the Web object is created, the HTTP headers and properties can be modified. Also, by default the HttpWebRequest method is GET. At this point, nothing has hit the network because an object needs to be created in order to set properties before the request is initiated.
The next step is to issue the request and receive the response from the destination. This is done by calling the GetResponse() method, which returns a WebResponse object that can be cast to an HttpWebResponse object if you need to access the HTTP-specific properties:
HttpWebResponse httpResponse;
httpResponse = (HttpWebResponse) httpRequest.GetResponse();
Dim httpResponse As HttpWebResponse
httpResponse = httpRequest.GetResponse()
The returned HttpWebResponse object will indicate the response code of the request via the StatusCode property, as well as a text description of the result in StatusDescription. If an entity is to be sent following the request, the next step is to call GetResponseStream(), which returns a Stream object that is then used to receive the entity data. Refer to Chapter 2 for information on manipulating Stream objects. If no response stream is associated with the HttpWebResponse, a ProtocolViolationException is thrown when GetResponseStream() is called.
Finally, once all data is received from the response stream, the stream should be closed either by calling the Stream.Close() method or by calling HttpWebResponse.Close(). Note that both methods can be called, but at least one must be invoked to free the connection. The HttpResponse.Close() method essentially calls the Close() method of the stream and is provided as a convenience. If neither is called, the application will leak resources and run out of allowed connections. Notice that since HTTP 1.1 is implemented, keepalives are enabled by default.
As shown, setting up an HTTP Web request and receiving the response is easy and straightforward; however, the previous example didn’t handle any exceptions. Exceptions are most commonly encountered when retrieving the response for a Web request and these exceptions take the form of an instance of a WebException class. Once a WebException occurs, the Status property indicates the type of failure. Common failures include underlying protocol failure, request timeout, etc. The Status property is WebExceptionStatus-enumerated type. Table 10-3 lists typical failures and their description, but consults the .NET Framework SDK for a complete description of each of the enumerated error types.
Table 10-3: WebExceptionStatus Members
|
|
Error |
Description |
ConnectFailure |
An error occurred at the transport (e.g., TCP) level. |
ConnectionClosed |
The connection was unexpectedly closed. |
Pending |
The asynchronous request is pending. |
ProtocolError |
A response was received but a protocol-level error occurred (see Table 10-4 for HTTP protocol-level errors). |
ReceiveFailure |
An error occurred receiving the response. |
RequestCanceled |
The request was canceled by the Abort() method. This error is used for any error not classified by the WebExceptionStatus enumeration. |
SendFailure |
An error occurred sending the request to the server. |
Success |
The operation completed successfully. |
Timeout |
No response was received during the specified timeout period. |
TrustFailure |
The server certificate could not be validated. |
If the indicated exception status is WebExceptionStatus.ProtocolError, which means a response from the server was retrieved but a protocol error was detected, then an HttpWebResponse object has been created and can be accessed to obtain more information about the failure. In the case of the HTTP protocol, the HTTP-specific error code can be retrieved from the HttpWebResponse class as the HttpStatusCode property. Table 10-4 lists the common HTTP return codes and their descriptions. Again, consult the .NET Framework SDK for a complete list of the errors and meanings.
Table 10-4: HttpStatusCode Members
|
||
Status Code |
HTTP Error Code |
Description |
Accepted |
202 |
The request has been accepted for further processing. |
BadRequest |
400 |
The server could not understand the request. This error is also used for any errors not classified by the HttpStatusCode enumeration. |
Continue |
100 |
The client can continue with the request. |
Forbidden |
403 |
The server has refused to respond to the request. |
GatewayTimeout |
504 |
An intermediate proxy server timed out while waiting for a response. |
InternalServerError |
500 |
A generic error occurred on the server while responding to the request. |
NotFound |
404 |
The requested resource does not exist on the server. |
OK |
200 |
The request was successful and the requested information was sent. |
ProxyAuthenticationRequired |
407 |
The proxy server requires an authentication header. |
RequestTimeout |
408 |
The client did not send the request in the time expected by the server. |
ServiceUnavailable |
503 |
The server cannot handle the request due to high load or because it is down. |
Unauthorized |
401 |
The requested resource requires authentication to access. |
The following example verifies in the catch block whether a response was returned and, if so, prints the HTTP status code information:
|
C#
HttpWebRequest httpRequest; HttpWebResponse httpResponse;
try { httpRequest = (HttpWebRequest) WebRequest.Create("http://www.winisp.net/goodrich"); httpResponse = (HttpWebResponse) httpRequest.GetResponse(); } catch ( WebException wex ) { if ( wex.Status == WebExceptionStatus.ProtocolError ) { httpResponse = (HttpWebResponse) wex.Response; Console.WriteLine("HTTP Response Code: {0}", httpResponse.StatusCode.ToString()); httpResponse.Close(); } } |
Dim httpRequest As HttpWebRequest
Dim httpResponse As HttpWebResponse
Try
httpRequest = WebRequest.Create("http://www.winisp.net/goodrich" )
httpResponse = httpRequest.GetResponse()
Catch wex As WebException
If wex.Status = WebExceptionStatus.ProtocolError Then
httpResponse = wex.Response
Console.WriteLine("HTTP Response Code: {0}", httpResponse.StatusCode.ToString())
httpResponse.Close()
End If
End Try
Now that we’ve covered the basics for HTTP Web requests and responses, we’ll move on to discuss some common scenarios in more detail. In this section, we present detailed samples for making HTTP requests with the GET and POST methods.
GET
GET retrieves content from Web servers and is the default method for most Web transactions. In the previous examples we showed how to construct instances of the HttpWebRequest class and how to retrieve the response header, but not how to retrieve the entity. The basic steps for issuing a GET request and handling the response are:
The following code sample illustrates these steps. Once it obtains the HttpWebResponse object, it retrieves the stream handle and reads the returned data until the end. If the retrieved entity is text, a StreamReader is created to retrieve the data; otherwise, if the entity is an image, a BinaryReader is used.
HttpWebRequest httpRequest = null;
HttpWebResponse httpResponse = null;
BinaryReader binReader = null;
StreamReader streamReader = null;
try
{
// Create the HTTP request object
httpRequest = (HttpWebRequest) WebRequest.Create("http://www.winisp.net/goodrich");
// Set some HTTP specific headers
httpRequest.UserAgent = "My User Agent/1.0";
// Get the response object
httpResponse = (HttpWebResponse) httpRequest.GetResponse();
if ( httpResponse.ContentType.StartsWith( @"image" ) )
{
// For image entities, use the binary reader
binReader = new BinaryReader(httpResponse.GetResponseStream() );
byte [ ] responseBytes;
// Read the response in 4KB chunks
while ( true )
{
responseBytes = binReader.ReadBytes( 4096 );
if ( responseBytes.Length == 0 )
break;
// Do something with the data
}
}
else if ( httpResponse.ContentType.StartsWith( @"text" ) )
{
// For text entities, use the text reader.
streamReader = new StreamReader(httpResponse.GetResponseStream(), Encoding.UTF8 );
string httpContent = streamReader.ReadToEnd();
// Do something with the data
}
}
catch ( WebException wex )
{
Console.WriteLine("Exception occurred on request: {0}", wex.Message );
if ( wex.Status == WebExceptionStatus.ProtocolError )
httpResponse = wex.Response;
}
finally
{
if ( httpResponse != null )
httpResponse.Close();
if ( binReader != null )
binReader.Close();
if ( streamReader != null )
streamReader.Close();
}
Dim httpRequest As HttpWebRequest = Nothing
Dim httpResponse As HttpWebResponse = Nothing
Dim binReader As BinaryReader = Nothing
Dim streamReader As StreamReader = Nothing
Try
' Create the HTTP request object
httpRequest = WebRequest.Create("http://www.winisp.net/goodrich")
' Set some HTTP specific headers
httpRequest.UserAgent = "My User Agent/1.0"
' Get the response object
httpResponse = httpRequest.GetResponse()
If httpResponse.ContentType.StartsWith("image") Then
' For image entities, use the binary reader
binReader = New BinaryReader(httpResponse.GetResponseStream())
Dim responseBytes() As Byte
' Read the response in 4KB chunks
While (True)
responseBytes = binReader.ReadBytes(4096)
If responseBytes.Length = 0 Then
GoTo AfterLoop
End If
' Do something with the data
End While
AfterLoop:
ElseIf httpResponse.ContentType.StartsWith("text") Then
' For text entities, use the text reader.
streamReader = New StreamReader(httpResponse.GetResponseStream(), Encoding.UTF8)
Dim httpContent As String = streamReader.ReadToEnd()
' Do something with the data
End If
Catch wex As WebException
Console.WriteLine("Exception occurred on request: {0}", wex.Message)
If wex.Status = WebExceptionStatus.ProtocolError Then
httpResponse = wex.Response
End If
Finally
If Not httpResponse Is Nothing Then
httpResponse.Close()
End If
If Not binReader Is Nothing Then
binReader.Close()
End If
If Not streamReader Is Nothing Then
streamReader.Close()
End If
End Try