Title:Weighing the World Wide Web: The art and science of load testing for unique high-spike conditions
Authors:Radha Nagaraja
Publication:MW18: Museums and the Web 2018

The website of an organization is its global public face in the modern digital age, and smooth functioning of the website is key to meeting an organization's mission. The appropriate Web server infrastructure and configuration are, by common sense wisdom, a function of the expected traffic patterns and the content served on the website. However, a challenging wrinkle in Web server provisioning is posed by high spikes in traffic due to specific unique events. A particularly dramatic instance of this challenge is faced annually by the 9/11 Memorial and Museum on September 11th, when website traffic increases by 35x To ensure smooth functioning of the website, the annual run-up to the expected high-spike day utilizes a myriad of tools, including best-estimate characterization of expected load; provisioning of a surrogate server infrastructure set up to best mimic the website configuration (and any special content/functionalities planned for the specific day when the high spike is expected) for the future high-spike time; incorporation or appropriate modeling of the caching and load balancing mechanisms expected to be utilized; user simulation and browser automation technologies to realistically mimic spike traffic patterns; and monitoring of quantitative metrics of website performance during load tests over adequate time intervals to uncover subtle issues that might manifest only under sustained load. Applying the set of strategies above has proven to be a time-tested and robust policy for the 9/11 Memorial and Museum to meet the huge spike in Web traffic on September 11th of each year. In this presentation, the strategies above will be covered in detail including overview of the techniques used to estimate future high-spike traffic, rig up surrogate servers, create realistic mock traffic, and best ensure that a website remains performant under high-spike conditions.