Drupal 6 CDN integration: a test case

published on February 17, 2010

In this final article in my bachelor thesis series, I explain how I proved that the work I had done for my bachelor thesis (which includes the Episodes module, the Episodes Server module, the CDN integration module and File Conveyor) actually had a positive impact on page loading performance. For that, I converted a fairly high-traffic web site to Drupal, installed File Conveyor to optimize & sync files to both a static file server and an FTP Push CDN, used the CDN integration module to serve files from either the static file server or the FTP Push CDN (the decision to pick either of those two is based on the visitor’s location, i.e. the IP address), measure the results using Episodes and prove the positive impact using Episodes Server’s charts.

Previously in this series:

As a back-up plan in case there would not be much feedback from companies (as turned out to be the case), I wanted to have a web site under my own control to use as a test case. That web site is driverpacks.net (see the first figure below for a screenshot of its homepage). It is the web site of an open source project, with more than 100,000 visits per month and more than 700,000 pageviews per month, with traffic coming from all around the world. These fairly large numbers and the geographical spread of its visitors make it a good test case for measuring the effect of a CDN. See the second figure below for a map and details.
Visitors come from 196 different countries, although the top three countries represent more than a quarter of the visitors and the top ten countries represent more than half of the visitors. Nevertheless, this is still a very geographically dispersed audience.

See the attached figures:

The goal was obviously to port this site to Drupal (which is not a part of this thesis of course) and to install the Episodes module. During about a week, statistics would be collected while no CDN was being used. Then, I would install the daemon on the server to sync the files to a CDN. Next, I would install the Drupal CDN integration module. Then again for about a week, statistics would be collected while the CDN was being used. Hopefully, by visualizing the collected episode measurements, it would be confirmed that this had indeed had a positive effect.

1. Implementation

The web site was previously a combination of static HTML and a bit of PHP. On June 11, I launched the new web site, which was built using Drupal. Drupal’s CSS and JavaScript aggregation options were enabled. Due to this, Drupal combines the CSS files for the same media type (screen, all …) and in the same section (header or footer) and the JavaScript file sin the same section (header or footer), thereby already significantly reducing the number of HTTP requests and page loading time. If I would have turned this on when enabling CDN integration, the effect would have been far more significant. But the comparison would have been unfair. Hence, these options were enabled, right from the beginning.
Also the JavaScript for the Google Analytics service has been cached on the web server since the launch of the web site, because the Google Analytics module for Drupal supports it. This makes it even harder to show the effect.
Finally, driverpacks.net is a site with virtually no images (because it slows down the page loading, but mostly because I am not a designer). Only two images are loaded for the web site’s style, one is loaded for each of the two statistics services and one is loaded for the ad at the top. Only the two images that are part of the web site’s style can be cached. This makes it again more difficult to show the effect.
This suggests that the results that will be observed should be magnified significantly for more media-rich web sites, which is typically the case nowadays. However, I did not have such a web site around to test with. It is likely that if the effect is noticeable on this web site, it is dramatically noticeable on a meda-rich web site.

On June 21 around 2 AM GMT+1, I enabled CDN integration for all users. To show the potential of the daemon in combination with the CDN integration module however, I implemented the cdn_advanced_pick_server() function that the Drupal CDN integration module calls when it is running in advanced mode and when that function exists. It really allows you to create any desired logic for routing users to specific CDNs or static file servers, as is demonstrated in the following code:

function driverpacksnet_map_country_to_server($country_code) {
  $countries_for_warp_server = continents_api_get_countries('EU') + array('RU');
  if (in_array($country_code, $countries_for_warp_server)) {
    return 'warp';
  }
  else {
    return 'simplecdn';
  }
}

/**
 * Implementation of cdn_advanced_pick_server().
 */
function cdn_advanced_pick_server($servers_for_file) {
  static $server_name;

  if (!isset($server_name)) {
    // Determine which server to use for the current visitor.
    $ip = $_SERVER['REMOTE_ADDR'];
    $country_code = ip2country_get_country($ip);
    $server_name = driverpacksnet_map_country_to_server($country_code);
  }

  // Try to pick the desired server - if the file being served is available on
  // our the desired server.
  foreach ($servers_for_file as $server_for_file) {
    if ($server_for_file['server'] == $server_name) {
      return $server_for_file;
    }
  }

  // In case our desired server does not have the file, pick the first server
  // that does have it.
  return $servers_for_file[0];
}

Both a static file server and a CDN are being used, but not just for demonstrative purposes. It was decided to use two servers because the latency to SimpleCDN’s servers is higher than acceptable in Europe (around 100 milliseconds or more). They claim to have servers in Amsterdam, but for some reason all requests from my IP (in Belgium) are being routed to a datacenter in Phoenix, Arizona in the United States, which explains the high latency (this was discovered using the traceroute command). This was worse than not using a CDN for countries with low-latency connections to Belgium. Since latency is the major problem in the loading of CSS, JavaScript and other files referenced by the HTML (because of the multitude of round trips due to HTTP requests), I measured this using the ping command. I mostly relied on the “just ping” web service to get ping times from all around the world, to get a global view.
If only SimpleCDN would have been used, the web site would have become significantly slower in Europe (also in Russia, but less dramatically) and it would have been very likely that no improvement could be observed from the collected statistics.

The static file server (labeled warp) server is located in Berchem, Belgium and uses the static.driverpacks.net domain name and is an Apache HTTP server instance (with far future Expires headers, gzip enabled and Last-Modified headers disabled) optimized for static file serving and the CDN (labeled simplecdn) server uses the cdn.driverpacks.net domain name. This is a DNS CNAME record pointing to e1h21.simplecdn.net (this server is for SimpleCDN’s “StormFront” service, which claims to have 9 POPs in the United States and 2 POPs in Europe). Files served from this domain have gzip enabled, far future Expires headers and no Last-Modified headers. There are other CDN services than SimpleCDN that provide POPs all over the world, as opposed to just two continents. They are typically far more expensive though.

The Drupal ip2country module is used to map an IP to a country code. This module uses the database maintained by ARIN, the American Registry for Internet Numbers. ARIN is one of the five official Regional Internet Registries (RIR) responsible for assigning IP addresses. It is claimed to be 98% accurate, which should be sufficiently accurate. Via the continents_api module (which I wrote as an addition to the countries_api module), all country codes for Europe are collected in an array. Russia’s country code is appended to this array. This way, all European countries plus Russia are assigned to the warp server. Visitors from all other countries are assigned to the simplecdn server. This implies that files are being synced through the daemon to both the warp server (using the symlink or copy transporter, see “File Conveyor: design”, the transporter_symlink_or_copy.py section) and to the simplecdn server (using the FTP transporter, see “File Conveyor: design”, the transporter_ftp.py section).

Visitors are informed on the homepage from which server they are getting their static files, as can be seen in the following figure.

See the attached figure: “Visitor location block”.

One day later, again around 2 AM GMT+1, I added some optimizations: from that moment, Google Analytics’ JavaScript code was being executed after the window.onload event. This caused the executing of JavaScript code and the loading of an invisible 1x1 image to be delayed until after the page was fully rendered, thereby speeding up the rendering of the page and thus the perceived page load time.
The CSS and JavaScript of the delicious.com bookmarking widget (see the bottom right of the screenshot of driverpacks.net) on the homepage were also altered to be served from driverpacks.net’s web server (and after syncing, these files were being served from either the static file server or the CDN). The AJAX request to delicious.com’s servers was being delayed until after the window.onload event, i.e. when the page was fully rendered. This is not annoying because the information in this widget is not critical.
Finally, I also modified the loading of the image necessary for the whos.amung.us statistics service. Normally, this image is simply referenced from within the HTML. However, now it is being detected on the server side if the visitor has JavaScript enabled (Drupal then sets a has_js cookie, which can be detected on the server side) and in that case, a piece of JavaScript is inserted instead that is executed after the window.onload event, which inserts the HTML that loads the image. CSS is used to consume the whitespace until the image is loaded (where otherwise the image would have been), to prevent annoying relayouting (perceived as “flicker” by the end user).

3. Collected statistics

About 100 MB worth of statistics had been logged. These were then imported on June 25 (using the Episodes Server module, see “Improving Drupal: Episodes integration”, section 4: “Episodes Server module: reports”), resulting in a database table of 642.4 MB. More than 2.7 million episodes were collected over more than 260,000 page views. See the figure below for details (note: in the figure, the times are in GMT, which explain the discrepancy in time span). All screenshots listed in this subsection are made from the Episodes Server module running on driverpacks.net.

See the attached figure: “Episodes analysis: overall”.

This means episode measurements have been collected since the beginning of June 11 until the beginning of June 25, resulting in exactly two weeks of data, with the last 4 days having CDN integration. This is in fact too short to see the full effect of caching kicking in. driverpacks.net’s visitors typically don’t visit the site daily, but weekly or monthly. Hence many visitors have an empty cache, which drives the measured durations up. Fortunately, the effect of the CDN on visitors with an empty cache is clearly visible. The effect of visitors with primed caches is also visible, but less explicit.

Now the interesting part begins. There are five charts in figure below. The first four display the average duration of the backend and frontend episodes per day, for a given territory of visitors. The first chart gives the results for all collected data, i.e. globally. The second chart is for Brazil, the third for the Netherlands and the fourth for the United States. As we have seen, the United States and Brazil represent the largest groups of visitors, so their results should closely match the average real-world performance there, which means they should be well represented in these charts. The Netherlands were chosen because a location very close to the ‘warp’ server was also needed. The Netherlands were chosen over Belgium because more visitors originate from there, making the results more representative. On average, the U.S. had about 2,800 page views per day, Brazil had about 1,500 page views per day and the Netherlands had about 350 daily page views.
The fifth and last chart compares the average duration of the frontend episodes per day of the three countries with the global averages.

See the attached figure: “Episodes analysis: page loading performance”.

As this bachelor thesis is not page rendering performance, but about page loading performance, the charted data for backend episodes can be ignored. I have no explanation for why it peaks at different times around the world.
The frontend episodes contain the durations of loading all CSS, JavaScript and images.

Looking at the frontend episode durations in the first four charts, it is clear that in the period from June 11 until June 21, the variability in page loading time was very large. This may suggest that measurements are not taken over enough page views from a global point of view, but in the case of the United States and Brazil, which both had more than 2,500 page views per day (which should be sufficient to cancel out anomalies), it suggests that the variability of network latency (due to distance or congestion) can be great. This can probably be explained by the fact that most files had to be retrieved from the web server in Belgium. Belgium is a long distance from Brazil and the United States and therefor there are more opportunities (at every hop along the route) for the latency to increase, which is possibly the reason for the large variability.

Starting June 21, which is the date when CDN integration was enabled, there is a slight increase in some countries, but globally, there is a slight decrease. This is to be expected, as the frequent visitors have to download the static files again, this time from the CDN (simplecdn server) or static file server (warp server). On June 22, when another batch of changes was applied and the cached CSS and JavaScript files were once again obsolete, there is a slight increase globally, but a significant drop in the Netherlands.
It is likely that this is because whereas Dutch visitors previously had to wait for as much as five round trips to the United States (three to the delicious.com servers, one to Google Analytics’ server and one to the additional statistics service’s server), that is now reduced to just three (the CSS and JavaScript files for the delicious.com widget are being served from the CDN or, in the case of the Netherlands, from the static file server in Belgium).
However, starting on June 23, there is a clear, worldwide, downward trend. Especially in Brazil and the United States, this trend is very prominently visible in the charts. It is less strong in the Netherlands, because they were getting their files already from a server with a fairly small latency. The reason it drops even further for the Netherlands likely is that the static file server is configured properly to maximize client-side caching (i.e. in the visitor’s browser), whereas before the CDN and static file server were being used, static files were being served using the default Apache settings.

Finally, there is the analysis of the duration of the episodes (see the figure below) themselves, regardless of country. Through this chart, it is possible to examine which episode have the greatest impact on the overal page loading time and would thus benefit the most from optimizations. Unfortunately, I did not have the time to create a chart that displays the evolution over time. So it is currently impossible to see how much a specific episode has improved over time.
However, it is very clear that the headerjs episode (which represents the episode of loading the JavaScript in the <head> element of the HTML document) is by far the episode with the longest duration, thus is the best candidate for optimization. Probably even after CDN integration, this episode will remain the longest.

See the attached figure: “Episodes analysis: episodes”.

4. Conclusion

Globally, there has been a strong decrease in the page loading time and thus a strong increase in page loading performance. And since this test was conducted on a media-poor web site, it is likely that the effect is dramatically more noticeable on a media-rich web site. Since the downward trend lasted two days, it seems likely that the page loading time will become less variable because the static files have been cached in the browser.

Especially for countries that are located far from Belgium, such as Brazil and the United States, variability seems to have decreased significantly. This suggests that adding more servers around the world, at strategic locations (for example in the case of driverpacks.net, Bangkok in Thailand is the city that generates the most visitors around the world according to Google Analytics), may be a very effective method for improving page loading performance. Depending on the web site, this strategy may achieve equal or better performance than a CDN with servers all around the world, at a fraction of the cost.

It is certain that every web site can benefit from an analysis like the one above. The strategy to optimize page loading performance is different for every web site.

This is a republished part of my bachelor thesis text, with thanks to Hasselt University for allowing me to republish it. This is section thirteen in the full text, in which it was called “Test case: DriverPacks.net” instead of “Drupal 6 CDN integration: a test case”.

Previously in this series:

Comments