Async site scraping in ASP.NET 4.5

gistfile1.cs

public class SiteScrape : HttpTaskAsyncHandler
{
    public override async Task ProcessRequestAsync(HttpContext context)
    {
        using (var http = new HttpClient())
        {
            var downloadTasks = new List<Task<string>> {
                http.GetStringAsync("http://bing.com"),
                http.GetStringAsync("http://google.com"),
                http.GetStringAsync("http://oredev.org"),
                http.GetStringAsync("http://microsoft.com"),
                http.GetStringAsync("http://asp.net"),
                http.GetStringAsync("http://signalr.net")
            };
            while (downloadTasks.Count > 0)
            {
                var completed = await Task.WhenAny(downloadTasks);
                downloadTasks.Remove(completed);
                context.Response.Write(completed.Result);
                await context.Response.FlushAsync();
            }
        }
    }
}

The goodness of async/await in ASP.NET 4.5. This asynchronously downloads the HTML from all the sites in parallel and as each one returns, asynchronously flushes it to the response, i.e. it can be flushing back HTML to the browser while still downloading HTML from the other sites, in parallel.

I'd have no hope of writing this in APM (Begin/End).

hotness

Just AMAZING... :D

I never knew about HttpTaskAsyncHandler until now. The fact that it allows you to do IsReusable="true" is awesome. Can't wait to benchmark this thing.

Thanks for posting!

Throw some linq in there while you're at it:

while(downloadTasks.Any()) { ... }

:)

Would this work with Response.BufferOutput = false; if you are getting a Stream not a string?