< Chap 10 TOC: HTTP and .NET | Main | C++ Http Get Request Program Example >

 


 

Chapter 10 Part 1:

HTTP with .NET

 

 

What do we have in this chapter 10 Part 1?

  1. Overview

  2. Web Class Basics

  3. HttpWebRequest and HttpWebResponse

  4. Web Exceptions

  5. Common Scenarios

  6. GET

 

 

 

Overview

 

In the previous two chapters we examined how to access the network protocols directly through the Socket class, which allows for precise control over what is sent and received on a socket. While the Socket class is powerful, it is also cumbersome to implement higher-level protocols, such as Hypertext Transfer Protocol (HTTP), because the application has to handle the network transactions as well as build and parse the HTTP commands, which can be quite complex. This chapter introduces the HTTP-related classes, an application-level protocol built on top of TCP for accessing Web resources. The .NET Framework Web classes encapsulate the TCP details for retrieving Web resources into simple and easy- to-use classes.

This chapter is divided into several parts. First, we’ll introduce the basics of accessing HTTP-based Web resources via the HttpWebRequest and HttpWebResponse classes. This will be followed by the WebRequest and WebResponse classes that expose a uniform method of accessing any network resource based on request-response, such as files and HTML that can be extended to support other application-level protocols, including FTP (FtpWebRequest and FtpWebResponse) and file sharing.

Following that, we’ll cover more advanced topics such as the asynchronous model for accessing Web resources, as well as connection management, establishing secure connections over a Secure Sockets Layer (SSL), and application models that use Web-based classes from different areas (ASP.NET, console applications, etc.). We’ll finish the discussion with code-access security considerations when using the Web classes.

 

Web Class Basics

 

There are four classes of interest for accessing Web resources: HttpWebRequest, HttpWebResponse, WebRequest, and WebResponse. The first two are used specifically for HTTP communication, while the second two offer an extensible interface to a variety of application protocols, including HTTP.

The Web classes in this chapter are all request-response transactions. The client makes a request to a server to perform some action. The server receives the request and generates a response that is returned to the client. Following the response, the requested entity or content is transmitted.

The HTTP protocol defines several actions a client can request of the server. These actions are listed in Table 10-1. The most common method, GET, requests to retrieve a URI resource. For example if a GET request is made for http://www.microsoft.com/ an HTTP request is generated and sent to www.microsoft.com. The server then examines the request, generates a response indicating success, and sends the entity, which is the default HTML page for the Web site. The second column of the table indicates which version of the HTTP protocol introduced the method.

 

Table 10-1: HTTP Request Methods

 

Method

Minimum Version

Description

GET

0.9

Retrieves a URI resource.

HEAD

1

Retrieves only the metadata of a URI resource.

POST

1

Submits a form or other data (such as a SOAP message)

PUT

1.1

Uploads a file.

DELETE

1.1

Deletes a URI resource.

TRACE

1.1

Traces proxy chains.

OPTIONS

1.1

Queries HTTP server options.

 

Additionally, the HTTP protocol defines a series of header options that affect the behavior of the response. These options are simple string name/value pairs. The HTTP protocol defines numerous header options. The majority of these HTTP headers are exposed as properties of the HttpWebRequest, HttpWebResponse, WebRequest, and WebResponse classes. The ContentLength property that maps to the "Content-Length:" header is an example. Notice that some headers are valid only for requests, others for response, some for both request and response, and others for entities only. Table 10-2 lists the HTTP headers exposed as properties.

In addition to many of the headers being exposed as properties, the property Headers returns a WebHeaderCollection object that contains the entire set of headers associated with the request or response. The Add() method of the WebHeaderCollection class can be used to set additional HTTP headers in the collection. Notice that the headers exposed directly as properties cannot be set using the WebHeaderCollection. Instead, they should be accessed through their strongly typed properties.

 

Table 10-2: HTTP Header Properties

 

Property

HTTP Header

Valid Method

Description

Accept

“Accept:”

Request

Specifies acceptable media types (e.g., text/html, image/jpeg, etc.)

Connection

“Connection:”

Request, Response

Controls how connections are handled (e.g., disable persistent connections)

ContentLength

“Content-length:”

Entity

Specifies the length of the returned entity in bytes

ContentType

“Content-type:”

Entity

Specifies the media type of the returned entity (e.g., text\html, image\jpeg, etc.)

Expect

“Expect:”

Request

Indicates expected behavior of client by the server

IfModifiedSince

“If-Modified-Since:”

Request

Used to verify cache is up-to- date

Referer

“Referer:”

Request

Specifies the URL of the document containing the reference to the requested URL

TransferEncoding

“Transfer-encoding:”

Request, Response

Specifies any transformations applied to content (e.g., chunked)

UserAgent

“User-Agent:”

Request

Specifies the client software issuing the request

 

Only the most relevant headers are covered here; for a full listing of the HTTP headers and their corresponding properties, consult the .NET Framework SDK and RFC 2616.

 

HttpWebRequest and HttpWebResponse

 

The HttpWebRequest and HttpWebResponse provide HTTP-specific implementations of the WebRequest and WebResponse classes (which are discussed later) and allow access to URI-based HTTP resources. These classes offer a complete implementation of the HTTP 1 and 1.1 protocols.

As we mentioned earlier, HTTP traffic is based on request-response, and the first step of accessing HTTP-based Web resources is to create an instance of an HttpWebRequest object that indicates a Web resource. This is done by calling the Create() method of the WebRequest class as in the following sample:

 

C#

 

HttpWebRequest  httpRequest;

httpRequest = (HttpWebRequest) WebRequest.Create("http://www.winisp.net/goodrich/default.htm");

 

Visual Basic .NET

 

Dim httpRequest As HttpWebRequest

httpRequest = WebRequest.Create("http://www.winisp.net/goodrich/default.htm")

 

The example creates an HttpWebRequest object that points to the URI http://www.winisp.net/goodrich/default.htm. Notice that the constructor for the HttpWebRequest class should not be called - only the WebRequest.Create() method. See the WebRequest and WebResponse section of this chapter for more details on the requirements of calling the WebRequest.Create() method. Once an instance of the Web object is created, the HTTP headers and properties can be modified. Also, by default the HttpWebRequest method is GET. At this point, nothing has hit the network because an object needs to be created in order to set properties before the request is initiated.

The next step is to issue the request and receive the response from the destination. This is done by calling the GetResponse() method, which returns a WebResponse object that can be cast to an HttpWebResponse object if you need to access the HTTP-specific properties:

 

C#

 

HttpWebResponse  httpResponse;

httpResponse = (HttpWebResponse) httpRequest.GetResponse();

 

Visual Basic .NET

 

Dim httpResponse As HttpWebResponse

httpResponse = httpRequest.GetResponse()

 

The returned HttpWebResponse object will indicate the response code of the request via the StatusCode property, as well as a text description of the result in StatusDescription. If an entity is to be sent following the request, the next step is to call GetResponseStream(), which returns a Stream object that is then used to receive the entity data. Refer to Chapter 2 for information on manipulating Stream objects. If no response stream is associated with the HttpWebResponse, a ProtocolViolationException is thrown when GetResponseStream() is called.

Finally, once all data is received from the response stream, the stream should be closed either by calling the Stream.Close() method or by calling HttpWebResponse.Close(). Note that both methods can be called, but at least one must be invoked to free the connection. The HttpResponse.Close() method essentially calls the Close() method of the stream and is provided as a convenience. If neither is called, the application will leak resources and run out of allowed connections. Notice that since HTTP 1.1 is implemented, keepalives are enabled by default.

 

Web Exceptions

 

As shown, setting up an HTTP Web request and receiving the response is easy and straightforward; however, the previous example didn’t handle any exceptions. Exceptions are most commonly encountered when retrieving the response for a Web request and these exceptions take the form of an instance of a WebException class. Once a WebException occurs, the Status property indicates the type of failure. Common failures include underlying protocol failure, request timeout, etc. The Status property is WebExceptionStatus-enumerated type. Table 10-3 lists typical failures and their description, but consults the .NET Framework SDK for a complete description of each of the enumerated error types.

 

Table 10-3: WebExceptionStatus Members

 

Error

Description

ConnectFailure

An error occurred at the transport (e.g., TCP) level.

ConnectionClosed

The connection was unexpectedly closed.

Pending

The asynchronous request is pending.

ProtocolError

A response was received but a protocol-level error occurred (see Table 10-4 for HTTP protocol-level errors).

ReceiveFailure

An error occurred receiving the response.

RequestCanceled

The request was canceled by the Abort() method. This error is used for any error not classified by the WebExceptionStatus enumeration.

SendFailure

An error occurred sending the request to the server.

Success

The operation completed successfully.

Timeout

No response was received during the specified timeout period.

TrustFailure

The server certificate could not be validated.

 

If the indicated exception status is WebExceptionStatus.ProtocolError, which means a response from the server was retrieved but a protocol error was detected, then an HttpWebResponse object has been created and can be accessed to obtain more information about the failure. In the case of the HTTP protocol, the HTTP-specific error code can be retrieved from the HttpWebResponse class as the HttpStatusCode property. Table 10-4 lists the common HTTP return codes and their descriptions. Again, consult the .NET Framework SDK for a complete list of the errors and meanings.

 

Table 10-4: HttpStatusCode Members

 

Status Code

HTTP Error Code

Description

Accepted

202

The request has been accepted for further processing.

BadRequest

400

The server could not understand the request. This error is also used for any errors not classified by the HttpStatusCode enumeration.

Continue

100

The client can continue with the request.

Forbidden

403

The server has refused to respond to the request.

GatewayTimeout

504

An intermediate proxy server timed out while waiting for a response.

InternalServerError

500

A generic error occurred on the server while responding to the request.

NotFound

404

The requested resource does not exist on the server.

OK

200

The request was successful and the requested information was sent.

ProxyAuthenticationRequired

407

The proxy server requires an authentication header.

RequestTimeout

408

The client did not send the request in the time expected by the server.

ServiceUnavailable

503

The server cannot handle the request due to high load or because it is down.

Unauthorized

401

The requested resource requires authentication to access.

 

The following example verifies in the catch block whether a response was returned and, if so, prints the HTTP status code information:

 

 

C#

 

HttpWebRequest   httpRequest;

HttpWebResponse  httpResponse;

 

try

{

    httpRequest = (HttpWebRequest) WebRequest.Create("http://www.winisp.net/goodrich");

    httpResponse = (HttpWebResponse) httpRequest.GetResponse();

}

catch ( WebException wex )

{

    if ( wex.Status == WebExceptionStatus.ProtocolError )

    {

        httpResponse = (HttpWebResponse) wex.Response;

        Console.WriteLine("HTTP Response Code: {0}", httpResponse.StatusCode.ToString());

        httpResponse.Close();

    }

}

 

Visual Basic .NET

 

Dim httpRequest As HttpWebRequest

Dim httpResponse As HttpWebResponse

 

Try

    httpRequest = WebRequest.Create("http://www.winisp.net/goodrich" )

    httpResponse = httpRequest.GetResponse()

Catch wex As WebException

    If wex.Status = WebExceptionStatus.ProtocolError Then

        httpResponse = wex.Response

        Console.WriteLine("HTTP Response Code: {0}", httpResponse.StatusCode.ToString())

        httpResponse.Close()

    End If

End Try

 

Common Scenarios

 

Now that we’ve covered the basics for HTTP Web requests and responses, we’ll move on to discuss some common scenarios in more detail. In this section, we present detailed samples for making HTTP requests with the GET and POST methods.

 

GET

 

GET retrieves content from Web servers and is the default method for most Web transactions. In the previous examples we showed how to construct instances of the HttpWebRequest class and how to retrieve the response header, but not how to retrieve the entity. The basic steps for issuing a GET request and handling the response are:

 

  1. Create an HttpWebRequest object with the URL to retrieve.
  2. Retrieve the HttpWebResponse object for the request.
  3. Retrieve the stream to receive the entity data.
  4. Receive the data until the end of the stream is reached.

 

The following code sample illustrates these steps. Once it obtains the HttpWebResponse object, it retrieves the stream handle and reads the returned data until the end. If the retrieved entity is text, a StreamReader is created to retrieve the data; otherwise, if the entity is an image, a BinaryReader is used.

 

C#

 

HttpWebRequest  httpRequest = null;

HttpWebResponse httpResponse = null;

BinaryReader    binReader = null;

StreamReader    streamReader = null;

 

try

{

    // Create the HTTP request object

    httpRequest = (HttpWebRequest) WebRequest.Create("http://www.winisp.net/goodrich");

    // Set some HTTP specific headers

    httpRequest.UserAgent = "My User Agent/1.0";

    // Get the response object

    httpResponse = (HttpWebResponse) httpRequest.GetResponse();

 

    if ( httpResponse.ContentType.StartsWith( @"image" ) )

    {

        // For image entities, use the binary reader

        binReader = new BinaryReader(httpResponse.GetResponseStream() );

 

        byte [ ] responseBytes;

 

        // Read the response in 4KB chunks

        while ( true )

        {

            responseBytes = binReader.ReadBytes( 4096 );

            if ( responseBytes.Length == 0 )

                break;

            // Do something with the data

        }

    }

    else if ( httpResponse.ContentType.StartsWith( @"text" ) )

    {

        // For text entities, use the text reader.

        streamReader = new StreamReader(httpResponse.GetResponseStream(), Encoding.UTF8 );

        string httpContent = streamReader.ReadToEnd();

        // Do something with the data

    }

}

catch ( WebException wex )

{

    Console.WriteLine("Exception occurred on request: {0}", wex.Message );

    if ( wex.Status == WebExceptionStatus.ProtocolError )

        httpResponse = wex.Response;

}

finally

{

    if ( httpResponse != null )

        httpResponse.Close();

    if ( binReader != null )

        binReader.Close();

    if ( streamReader != null )

        streamReader.Close();

}

 

 

 

Visual Basic .NET

 

Dim httpRequest As HttpWebRequest = Nothing

Dim httpResponse As HttpWebResponse = Nothing

Dim binReader As BinaryReader = Nothing

Dim streamReader As StreamReader = Nothing

 

Try

' Create the HTTP request object

    httpRequest = WebRequest.Create("http://www.winisp.net/goodrich")

' Set some HTTP specific headers

    httpRequest.UserAgent = "My User Agent/1.0"

' Get the response object

    httpResponse = httpRequest.GetResponse()

 

    If httpResponse.ContentType.StartsWith("image") Then

' For image entities, use the binary reader

        binReader = New BinaryReader(httpResponse.GetResponseStream())

 

Dim responseBytes() As Byte

 

' Read the response in 4KB chunks

        While (True)

            responseBytes = binReader.ReadBytes(4096)

            If responseBytes.Length = 0 Then

                GoTo AfterLoop

            End If

' Do something with the data

        End While

AfterLoop:

    ElseIf httpResponse.ContentType.StartsWith("text") Then

' For text entities, use the text reader.

        streamReader = New StreamReader(httpResponse.GetResponseStream(), Encoding.UTF8)

 

Dim httpContent As String = streamReader.ReadToEnd()

 

' Do something with the data

    End If

Catch wex As WebException

    Console.WriteLine("Exception occurred on request: {0}", wex.Message)

    If wex.Status = WebExceptionStatus.ProtocolError Then

        httpResponse = wex.Response

    End If

Finally

    If Not httpResponse Is Nothing Then

        httpResponse.Close()

    End If

    If Not binReader Is Nothing Then

        binReader.Close()

    End If

    If Not streamReader Is Nothing Then

        streamReader.Close()

    End If

End Try

 

 

 


 

< Chap 10 TOC: HTTP and .NET | Main | C++ Http Get Request Program Example >