Exploring Windows 2003 Security: IP Security

The most recent article of our Windows Server 2003 series covered some of the security-related improvements in the networking functionality of Windows Server 2003 platform. This installment continues that discussion, with a focus on IPSec technology. IPSec was first implemented by Microsoft (based on its joint development effort with Cisco) in Windows 2000. Its primary purpose was to provide the ability to transmit IP-based traffic in a secure manner between two IPSec enabled hosts, regardless of the level of protection offered on intermediate networks.

We detail the improvements made to IPSec on Windows Server 2003 as well as offer a general overview of the technology.

Before delving into our discussion of the improvements in the area of IPSec on Windows Server 2003 , we will overview the technology’s main principles and the way it operates. In essence, IPSec combines a number of different parameters that both identify IP traffic to be secured and specify security settings to be assigned to it:

• Security Protocols — Authentication Header (AH) and Encapsulating Security Payload (ESP): Authentication Header offers authentication (checking credentials of hosts participating in data exchange), integrity (verifying that content has not been modified in transit), and replay protection (preventing from capturing IP packets and re-transmitting them to impersonate original sender), but not encryption (ensuring data confidentiality), while Encapsulating Security Payload includes all of them. Encryption algorithms (assigned to ESP) and hash algorithms used for verifying data integrity (assigned to both AH and ESP) are configurable.
• Mode of Operation — Transport and Tunnel: In transport mode, only payload (the data portion) of the IP packet is secured (this is done by appending relevant security protocol information to the original packet). In tunnel mode, the entire IP packet (including the header of original IP packet) is protected (this is done by encapsulating the original packet into a new IP packet). Transport mode is used much more commonly, since it constitutes the easiest way to provide VPN-based end-to-end secure communication between two computers (obviously both systems must support L2TP/IPSec). Tunnel mode is used primarily for communication between systems where L2TP/IPSec is not fully supported, so a tunnel is created to secure a portion of communication channel (typically to a gateway or end point that does not support L2TP/IPSec). Tunneling also requires static IP addresses at both endpoints and does not support filtering of IP traffic based on protocol or port number.
• Security Association: A combination of parameters, established using Internet Key Exchange (IKE) protocol, determine connection security. These parameters consist of common security protocols, authentication methods, and mode of operation agreed on by both hosts participating in the IPSec connection. IKE negotiates these parameters in two phases. The primary goal of the first one, called main mode, is to generate master key, which is used for encrypting the second phase (and for authenticating two connection partners until the second phase is completed). During the second phase, the master key secures negotiation between two hosts, resulting in the selection of a mutual security protocol, encryption method, and hash algorithm for the fully secured connection. This phase also results in the creation of a session key with configurable expiration parameters. Based on their value, a new session key is periodically re-created.

  Note that the IKE traffic is exempt from IPSec filtering (understandably so, since it is required to establish secure communication before the IPSec session is started). However, by default, this is the only exception of this kind in Windows Server 2003. In contrast, in Windows 2000 and XP, IPSec filtering exemption was less strict and also allowed all broadcast, multicast, Kerberos, and Resource Reservation Protocol (RSVP) traffic. Detailed information about modifying this default is provided in Microsoft Knowledge Base article 810207.

• Authentication Methods Used to Verify the Identity of Hosts During IKE Negotiation to Establish Security Association: Kerberos (limited to Active Directory, relies on forwarding computer credentials to Windows 200x domain controllers), certificates (requires a trusted Certificate Authority), or preshared keys (matching string of characters assigned to each computer). Note that preshared key authentication is not recommended due to its inherent weaknesses (e.g., such keys are stored in plaintext), and it should be used only if implementation of the remaining two methods is not possible.
• IP Filters: These filters specify protocol, port number, and source and destination of IP addresses to which all security parameters listed above will apply. This enables the system administrator to be selective when specifying what type of IP traffic should be secured.
• IP Filter Actions: The filter actions determine what type of action is triggered when traffic matching the IP Filter criteria is detected. Actions can belong to one of three general categories — permit, block, and negotiate (where the last one can be further configured by selecting security methods allowed to be negotiated).

IPSec policies are configured and stored as part of local and Active Directory group policies (although Windows Server 2003 also provides an option to use a persistent store for the location of locally assigned IPSec policy, independent of group policies. (This is accomplished with the NETSH command line utility, as described later in this article.) In either case, there are three pre-configured IPSec policies — Client (respond only), Server (request security), and Secure Server (require security), listed in the order of increasing security level. Creation of new ones is simplified by IP Security Policy Wizard, with associated sub-wizards (e.g., IP Security Rule Wizard, IP Filter Wizard, and IP Security Filter Action Wizard); however, even with help from the wizards, a number of available configuration settings can be initially overwhelming. After a policy is defined, it must be assigned (by selecting the Assign option from the context-sensitive menu of a selected IPSec policy) to become effective. Obviously, in case of a local policy, assignment applies to the local system, while in case of an Active Directory group policy, its impact depends on a container to which the policy is linked (as well as Security Group and WMI filtering settings).