Down Memory Lane
Memory tuning can be both science and art, as it covers a great deal of territory especially when dealing with Unix systems. Memory utilization affects swap space, network buffering, application and OS memory usage and caching. Each one of these areas can impact server performance substantially. The most difficult part of memory tuning is understanding which process, or processes, are responsible for the consumption of memory.
Unix programs such as swap, sar, ps, and top can help you identify the memory usage requirements for your OS and applications. On Windows NT, the included Performance Monitor is an excellent tool for performance analysis. The primary rule is to remember that you can never have too much memory. If you are running a Unix system, you must minimize the paging of memory to disk, since server performance will suffer when paged memory is used.
In addition to application and OS memory usage, make sure that you retain sufficient memory for network usage. Your servers are very dependent upon memory to be used as network buffers, which are responsible for the receipt and delivery of data to your server. Network buffer requirements can be identified by running netstat -m on Unix hosts. Pay close attention to any indications of memory requests being denied or delayed, which would indicate insufficient memory. It may be more efficient to decrease the size of the buffers servicing requests, as much of the HTTP traffic normally found at Web sites is considerably smaller than typical default buffer sizes.
Another network memory issue that merits notice is the amount of memory used for reverse DNS lookups when logging host names. You can save a considerable amount of memory and network usage by turning off DNS lookups, and possibly reassigning it to another machine that processes the log files offline and does the reverse resolution at that time.
Stuffing the Pipe
Network performance is the hardest component to measure consistently, which should come as no surprise when you consider how much the Internet changes. In the same vein as memory, there is no substitute for more capacity when it is needed.
One of the best tuning tricks for Unix servers involves increasing the size of the listening queues for incoming TCP connections. This comes about due to the long delays that are inherent with modem-based connections; your typical user is probably visiting your site at 28.8 Kbps or slower, and due to the amount of time it takes your server to service a request, it needs to maintain a longer queue length to hold onto the other incoming requests that must be responded to.
As mentioned earlier, it is sometimes beneficial to modify the number of buffers, as well as the buffer size, to improve performance. I usually prefer to make sure I have sufficient memory before I attempt to tweak buffer sizes. For Unix, the netstat -s command can give a wealth of information. Windows NT will also give you usable output with netstat, but not quite the detail that you will see on most Unix systems.
As we undergo growing pains in the Internet infrastructure, you will find that special attention has to be placed on the number of open connections your server maintains. TCP connections start and finish with a 3-way handshake. Unfortunately (due to the fact that many connections are broken off before they are properly closed), you may find that your machine is maintaining a large number of broken connections, which is, in effect, limiting the number of incoming connections that could be serviced. Netstat will likely show these connections in a FIN-WAIT2 state.
If you do see an inordinately high number of open connections, you should consult your documentation to see if the default value for TCP timeout can be changed. Be sure not to set it too low if the machine is to be used for other purposes.
Hopefully, this column has given you a few things to think about. Let me throw in a couple of things that transpired in the real world when putting the new servers online at WRQ. After we migrated our content and programming to the new box, we did in-house testing to establish benchmarks and to tune as needed. Although our intentions were good, this was a less than accurate method. We never exposed the server to the latency and connection difficulties that are inherent with the Internet. Accordingly, our benchmarks and tuning were subject to some rapid changes, to the point that we actually changed our server software to achieve the performance we needed.
Tuning is definitely a fine art that takes quite a while to get the hang of. If you have any additional insights or comments, feel free to reach me at jayl@digitalrain.net.