WebServer Compare: The Yin and Yang of Benchmarks

“Benchmarking software, including Web servers, offers a tantalizing paradox: Measurements are both concrete and tangible, yet at
the same time vague and ambiguous. Benchmarks hand us hard numbers, but what do these numbers mean? Often very little, in a
broad sense, or sometimes something very specific in a narrow sense.”

>The upshot of all this is that Web server benchmarks must be considered in a very limited manner. A benchmark for a server
running on a particular operating system, on a particular hardware configuration, focuses only on that specific setup. The further
the setup deviates from the tested habitat, the less meaningful the benchmark becomes.

“Web servers are not unlike zoo animals, quite reliant on their habitant. It’s simply unrealistic to consider the performance of a
specific Web server without regard to its operating environment — both its software environment, or operating system, and its
hardware environment, or the physical components that make up the machine. All of these factors can dramatically influence the
final numbers — how quickly and nimbly a Web server can respond and deliver to network requests.”

“What’s more, benchmarks can vary widely in how they measure a server. A server’s efficiency may also vary depending on the task
(e.g., whether its delivering static content to many simultaneous visitors or generating dynamic content).”

“The upshot of all this is that Web server benchmarks must be considered in a very limited manner. A benchmark for a server
running on a particular operating system, on a particular hardware configuration, focuses only on that specific setup. The further
the setup deviates from the tested habitat, the less meaningful the benchmark becomes.”