|
Connection Management
When retrieving multiple URIs, it is often the case that many of the resources reside on the same host. If an application needs to retrieve hundreds or thousands of URIs, the application should limit the number of concurrent HTTP requests. This allows efficient use of network bandwidth and lessens the load on the Web server. To accomplish this, the .NET Framework offers the ServicePointManager and ServicePoint classes for managing HTTP requests. The ServicePointManager class defines limits for a destination, such as the maximum number of concurrent connections allowed at any given time. The first step for managing Web connections is to establish the limits on the ServicePointManager that will be applied to subsequent connection requests. Table 10-5 lists the important properties of the ServicePointManager class.
Once the restrictions are set on the ServicePointManager, a ServicePoint object is retrieved for a given server before any Web requests are issued to that server. A ServicePoint is retrieved by calling FindServicePoint(). If a ServicePoint already exists from a previous request, the same object is returned; otherwise, a new ServicePoint is created so long as the MaxServicePoints limit is not exceeded. The following code illustrates this:
C#
ServicePoint sp;
GlobalProxySelection.Select = new WebProxy("http://myproxy:80/");
ServicePointManager.MaxServicePoints = 2; ServicePointManager.DefaultConnectionLimit = 2; ServicePointManager.MaxServicePointIdleTime = 20000; // 20 seconds
sp = ServicePointManager.FindServicePoint(new Uri("http://www.one.com")); // GetUriResource retrieves a URI GetUriResource("http://www.one.com/one.html");
sp = ServicePointManager.FindServicePoint(new Uri("http://www.two.com")); GetUriResource("http://www.two.com/two.html") |
Dim sp As ServicePoint
GlobalProxySelection.Select = New WebProxy("http://myproxy:80/")
ServicePointManager.MaxServicePoints = 2
ServicePointManager.DefaultConnectionLimit = 2
ServicePointManager.MaxServicePointIdleTime = 20000 ’ 20 seconds
sp = ServicePointManager.FindServicePoint(New Uri("http://www.one.com"))
' GetUriResource retrieves a URI
GetUriResource("http://www.one.com/one.html")
sp = ServicePointManager.FindServicePoint(New Uri("http://www.two.com"))
GetUriResource("http://www.two.com/two.html");
In this example, the application establishes a limit of two concurrent servers (MaxServicePoints) with two concurrent requests to each sever (DefaultConnectionLimit). It then sets www.one.com as the first ServicePoint to be managed and www.two.com as the second managed site. The GetUriResource() function issues an asynchronous HttpWebRequest for the resource.
The service point limit in the example is two, and only two concurrent ServicePoint objects are requested; therefore, the subsequent Web requests succeed. However, if a third call to FindServicePoint was requested to a server (for example, www.three.com) that had not been requested previously, an InvalidOperationException would be thrown because this third ServicePoint would exceed the established limit.
Only a single Web request is issued for each ServicePoint. If three concurrent instances of HttpGetRequest are issued for URIs residing on a single ServicePoint host (such as www.one.com), the first two are allowed while the third is blocked until one of the initial requests completes.
Create a new CLR console application project and you might want to use ServicePointCP as the project and the solution names.
Add the following code that includes two classes and main().
using namespace System; using namespace System::Net; using namespace System::Collections; using namespace System::Threading; using namespace System::IO;
/// <summary> /// Maintains context information for each HTTP request /// </summary> public ref class HttpState { public: HttpWebRequest^ httpRequest; HttpWebResponse^ httpResponse; Stream^ httpResponseStream; Uri^ uriResource; array<Byte>^ readBuffer; ManualResetEvent^ doneEvent; int bytesReceived;
public: static const int DefaultReadBufferLength = 4096;
/// <summary> /// Constructor for initializing HttpState object /// </summary> /// <param name="resource">URI to retrieve</param> /// <param name="done">Event to signal when done</param> public: HttpState(Uri^ resource, ManualResetEvent^ done) { httpRequest = nullptr; httpResponse = nullptr; uriResource = resource; readBuffer = gcnew array<Byte>(DefaultReadBufferLength); doneEvent = done; bytesReceived = 0; } };
/// <summary> /// Contains the ServicePoint sample /// </summary> public ref class ServicePointClass { /// <summary> /// Displays usage information /// </summary> public: static void usage() { Console::WriteLine("Executable_file_name [-u URI] [-p proxy] [-l count] [-s count]"); Console::WriteLine("Available options:"); Console::WriteLine(" -u URI URI resource to retrieve"); Console::WriteLine(" -p proxy Proxy server to use"); Console::WriteLine(" -l count Maximum number of concurrent connections"); Console::WriteLine(" -s count Maximum number of ServicePoint instances"); Console::WriteLine(); }
/// <summary> /// Asynchronous delegate for read operation on the HTTP stream. /// </summary> /// <param name="ar">Asynchronous context information for operation</param> public: static void HttpStreamReadCallback(IAsyncResult^ ar) { HttpState^ httpInfo = (HttpState^)ar->AsyncState;
try { long percent = 0; int count = httpInfo->httpResponseStream->EndRead(ar);
httpInfo->bytesReceived += count; percent = (long)(httpInfo->bytesReceived * 100) / (long)httpInfo->httpResponse->ContentLength; Console::WriteLine("{0}: read {1} bytes: {2}%", httpInfo->uriResource->AbsoluteUri->ToString(), count, percent.ToString()->PadLeft(3));
if (count > 0) { PostStreamRead(httpInfo); } else { httpInfo->httpResponseStream->Close(); httpInfo->httpResponseStream = nullptr; httpInfo->httpResponse->Close(); httpInfo->httpResponse = nullptr; httpInfo->doneEvent->Set(); } } catch (Exception^ ex) { Console::WriteLine("Exception occurred: {0}", ex->Message); if (httpInfo->httpResponseStream != nullptr) httpInfo->httpResponseStream->Close(); if (httpInfo->httpResponse != nullptr) httpInfo->httpResponse->Close(); httpInfo->doneEvent->Set(); } }
/// <summary> /// Post as an asynchronous receive operation on the HTTP stream. /// </summary> /// <param name="httpInfo">Context information for HTTP request</param> public: static void PostStreamRead(HttpState^ httpInfo) { IAsyncResult^ asyncStreamRead = httpInfo->httpResponseStream->BeginRead( httpInfo->readBuffer, 0, httpInfo->readBuffer->Length, gcnew AsyncCallback(HttpStreamReadCallback), httpInfo ); }
/// <summary> /// Asynchronous delegate for the HTTP response. /// </summary> /// <param name="ar">Asynchronous context information</param> public: static void HttpResponseCallback(IAsyncResult^ ar) { HttpState^ httpInfo = (HttpState^)ar->AsyncState;
try { httpInfo->httpResponse = (HttpWebResponse^)httpInfo->httpRequest->EndGetResponse(ar); httpInfo->httpResponseStream = httpInfo->httpResponse->GetResponseStream(); PostStreamRead(httpInfo); } catch (WebException^ wex) { Console::WriteLine("Exception occurred: {0}", wex->Message); Console::WriteLine(wex->StackTrace); httpInfo->doneEvent->Set(); } }
/// <summary> /// Retrieves the given resource /// </summary> /// <param name="uriResource">URI to retrieve</param> /// <param name="doneEvent">Event to signal when done</param> public: static void GetUriResource(Uri^ uriResource, ManualResetEvent^ doneEvent) { HttpState^ httpInfo = gcnew HttpState(uriResource, doneEvent);
try { Console::WriteLine("Issuing an async GET request for: {0}", uriResource->ToString()); httpInfo->httpRequest = (HttpWebRequest^)WebRequest::Create(uriResource->AbsoluteUri); httpInfo->httpRequest->BeginGetResponse(gcnew AsyncCallback(HttpResponseCallback), httpInfo); Console::WriteLine("POSTED lor!"); } catch (WebException^ wex) { Console::WriteLine("Exception occurred: {0}", wex->Message); doneEvent->Set(); } } };
/// <summary> /// Main application which parses the command line and issues the HTTP request. /// </summary> /// <param name="args">Command line arguments</param> int main(array<System::String ^> ^args) { ArrayList^ uriList = gcnew ArrayList(); IWebProxy^ httpProxy = WebRequest::DefaultWebProxy; int connectionLimit = 2, servicePointLimit = 1; int i;
// Parse the command line if(args->Length != 0) { for(i=0; i < args->Length ;i++) { try { if ( ( args[i][0] == '-' ) || ( args[i][0] == '/' ) ) { switch ( Char::ToLower( args[i][1] ) ) { case 'u': // URI to download (get) uriList->Add( gcnew Uri( args[ ++i ] ) ); break; case 'p': // Name of proxy server to use httpProxy = gcnew WebProxy( args[ ++i ] ); break; case 'l': // Number of connections limited to service point manager connectionLimit = Convert::ToInt32( args[ ++i ] ); break; case 's': // Service point limit servicePointLimit = Convert::ToInt32( args[ ++i ] ); break; default: ServicePointClass::usage(); return 0; } } } catch(Exception^ err) { Console::WriteLine("Error: " + err->Message); ServicePointClass::usage(); return 0; } } } else { ServicePointClass::usage(); return 0; }
try { array<ManualResetEvent^>^ requestEvents = gcnew array<ManualResetEvent^>(uriList->Count); // Setup the ServicePointManager first WebRequest::DefaultWebProxy = httpProxy; ServicePointManager::MaxServicePoints = servicePointLimit; ServicePointManager::DefaultConnectionLimit = connectionLimit; ServicePointManager::MaxServicePointIdleTime = 20000; // 20 seconds Console::WriteLine("Uri count: {0}", uriList->Count); for(i=0; i < uriList->Count ;i++) { Console::WriteLine("Uri: {0}", ((Uri^)uriList[i])->ToString() ); String^ baseUri = "http://" + ((Uri^)uriList[i])->Host + "/"; Console::WriteLine("Base URI: {0}", baseUri ); ServicePoint^ sp = ServicePointManager::FindServicePoint((Uri^)uriList[i], httpProxy ); requestEvents[i] = gcnew ManualResetEvent( false ); ServicePointClass::GetUriResource( (Uri^)uriList[i], requestEvents[i] ); } Console::WriteLine("Waiting..."); ManualResetEvent::WaitAll( requestEvents ); Console::WriteLine("Considered done..."); } catch ( Exception^ ex ) { Console::WriteLine("Exception occurred: {0}", ex->Message); } return 0; } |
Build and run the project. The following is the sample output.
The following is the sample output when run from the command prompt with two URIs. There is an error on the percentage of the progress for one of the URI. This is left for your exercise to rectify the logical error!