70-240 in 15 minutes a week: Windows 2000 IPSec

By ServerWatch Staff (Send Email)
Posted Dec 2, 2001


by Dan DiNicolo
http://www.2000trainers.com

Welcome to article number 27 in my 70-240 in 15 minutes a week series. This article covers IPSec and Certificate Services in Windows 2000. This includes a look at understanding, using and configuring IPSec policies and related settings, including those found in Group Policy. This article is the second-last in the 70-240 in 15 minutes a week series, and again falls under the networking services portion of the exam.

The material covered in this article includes:

- IPSec Overview
- Configuring IPSec Policies


IPSec Overview

IPSec is a security protocol included in the Windows 2000 TCP/IP stack. Its function is simple - to secure TCP/IP communications within or between networks according to the parameters and rules that you lay out. While the concept itself is simple enough, the actual implementation of policies that will do what you want actually requires a deeper understanding of TCP/IP. Not only must you understand how communication takes place on the network, but you must also be able to translate your security needs into specific explicit policies. That requires knowledge of protocols, port numbers, and more, as well as a plan for how and where you need to encrypt or confirm the integrity of traffic. While it might immediately seem like a good idea to encrypt all traffic on your network, that isn't necessarily practical or a good idea. Understand that you pay a price for encryption in the form of higher CPU utilization. This might not be a big deal for an individual workstation with a high-speed processor, but consider also the load on the server that it handling (and encrypting) multiple simultaneous encrypted connections. Instead, focus on using encryption where required for security purposes, creating a setup that meets the needs of your organization. It will seldom be a one-size-fits-all scenario.

If you are considering using IPSec, you first need to understand a bit about how it works. First of all, IPSec encryption / authentication differs from what you may already be familiar with in terms of security products. Most people have worked or are familiar with application-level encryption, where a program (such as PGP or similar) actually handles the encryption/decryption/signing processes. In contrast, IPSec actually resides in the protocol stack, independent of applications. Because of this, you are able to encrypt the data from any application prior to transmitting it over the network. Imagine that a client application on computer 1 wants to send encrypted data to a server application on computer 2. While there certainly needs to be some coordination between the two systems (we'll get to that in a bit), the applications on either side actually have nothing to do with the encryption process. The client application would simply begin passing data down the protocol stack like normal, where it would be 'intercepted' by IPSec, encrypted, and sent to the server system. On the server, the data moves up the stack, and is decrypted by IPSec prior to being passed to the application. For all intents and purposes, the encryption is transparent to the person on the client side - something very favorable indeed.

While many books and articles drown on about the various reasons for using encryption on a network, I'll spare you the details. Everyone who has worked with networks knows that if a data stream hasn't been secured, then it is theoretically possible for some intermediary person to possibly read or change the data. The various applications and attacks that make that possible are well documented elsewhere, and certainly some are more common than others. All things being equal, it is up to you to decide what is considered sensitive information on your network, and how you wish to proceed with policies that meet your needs. IPSec actually provides both encryption and authentication services, and understanding the difference is key to your planning process. 

IPSec is most commonly looked at as a protocol for securing network traffic via encryption. The way in which this is handled is via something called ESP, the Encapsulating Security Payload. The purpose of ESP in to provide data encryption - while packets are still visible on the network if you used a network sniffer or similar tool, the actual contents (often referred to as the payload, or less technically, the 'important stuff') of the packet beyond the original IP header are encrypted. How the encryption process happens I'll cover in a moment. The other function that IPSec is capable of is authentication and integrity settings via AH, the Authentication Header. That is, instead of actually encrypting a packet, it is rather signed, such that the receiver can be sure that the data came from the original sender and vice versa. The concern in this case is that the data hasn't been modified in transit, and that somebody (or some machine) isn't trying to impersonate someone else. While AH provides data integrity, it does nothing to actually secure the data stream, unless used in conjunction with ESP. Both AH and ESP can be used alone, or in conjunction with one another.

A small note here - be aware that ESP traffic uses IP protocol 50 to identify itself, while AH uses IP protocol 51. This is important in cases where you want these types of traffic to pass through a firewall or filtering device. Note also that IPSec will require that traffic can be passed over UDP port 500, over which ISAKMP/Oakley transfers (to be discussed shortly) take place.

Another important IPSec concept to consider is the mode in which you would have it participate on the network. The two IPSec modes are referred to as Transport mode and Tunnel mode. Transport mode allows for IPSec connections between 2 or more systems, used on demand, as the policy that you have created dictates. For example, you might specify that when any computer on the HR subnet attempts to communicate with the HR server, all traffic must be encrypted and signed. Or, you might specify that when any computer in the organization accesses the employee benefits database, all communication should be signed only. The level of granularity that you wish to use is up to you, just remember that the more complex your policies, the more difficult they may be to create and administer. Also remember that in order for a system to use IPSec in Transport mode, that system must use the IPSec protocol. Of the Microsoft operating systems, only Windows 2000 (and now XP) support these capabilities natively. Tunnel mode functions differently, and with a different purpose in mind. It creates a secured (and/or authenticated) tunnel between to endpoints. The idea is that traffic that passes through the tunnel in encrypted on one end and decrypted on the other, a scenario that would most commonly apply to moving between two private networks via an unsecured network such as the Internet. The benefits of using IPSec in Tunnel more are many. Firstly, it is relatively simple to set up (especially compared to Transport mode). Secondary, clients or servers on either side on the tunnel need not have any concept of IPSec or how it works. They simply send traffic destined for a system on the other side of the tunnel, and when it hits the first server the packets are encrypted, and subsequently decrypted on the other end. As you have probably considered, this does mean that the packets do travel over parts of both local networks in an unencrypted state. Whether that works for you will depend upon your specific security needs. Use a combination or tunnel mode and transport mode if it better meets your needs.

Many other documents also spend an incredible amount of time and energy explaining the internal processes that IPSec goes through in attempting to negotiate security. While this is certainly important, it is also easy to get bogged down in details and forget exactly what is going on. Essentially, if two computers want to use IPSec for the purpose of securing direct communication between them, they will each have to be configured with an IPSec policy. These policies are configured via local or group policy in Windows 2000, and will be discussed shortly. The policies need to work out the necessary details such that computers understand not only which protocols need to be secured, but also what key settings need to be used, authentication mechanisms, and so forth. The downside of IPSec is that there are many places where your configuration can be wrong. However, if you have everything mapped out properly in advance, the actual configuration should be fairly straightforward. Imagine if I have configured policies on two systems, and done it correctly. When the first system (say the client) attempts to communicate using IPSec with the second (say the server), they will first need to agree on a variety of factors including the key length to be used, security protocols, and something called the Security Parameter Index (SPI - this is what uniquely distinguishes one secured session from another). All of these things constitute something called a Security Association, the basis for the IPSec communication. You may have heard of something called the Internet Security Association and Key Management Protocol (ISAKMP) - this is the protocol used to actually establish security associations. There are actually two phases of security association negotiation, Phase I and Phase II. Phase I is responsible for the initial setup of a secure and authenticated channel between the two systems. Phase II is responsible for the actual security and negotiation of the data transfer, and the process of refreshing session keys used during the process. At the end of the Phase I SA lifetime (which is configurable), any Phase II SAs that exist may still exist, as they have their own lifetimes. However, if a Phase II data transfer were received without a Phase I SA in place, data transfer would be rejected. 

Before getting into the actual configuration of IPSec policies, you should also have an awareness of a key determination protocol called OAKLEY. Basically, OAKLEY is the one on each computer that works in conjunction with ISAKMP and is responsible for generating and managing the authenticated keys that are ultimately used to secure IPSec communications. 
Article 27 in Dan DiNicolo's 70-240 in 15 minutes a week series covers IPSec and Certificate Services in Windows 2000. This includes a look at understanding, using and configuring IPSec policies and related settings, including those found in Group Policy.

Configuring IPSec Policies


Now that that's all settled, lets take a look at actually setting up IPSec policies. Actually, before we get there, you should have an awareness of the three IPSec policies that exist by default, and what they means. Found in the Computer Configuration section of group policy under \Windows Settings\Security Settings\IPSec Policies, the there policies that exist are Client (Respond Only), Secure Server (Require Security), and Server (Request Security). Note that none of the policies are assigned by default, and that only one IPSec policy can be assigned on a system at any given point in time. The there policies are discussed after the screen shot below:

Client (Respond Only) - As the name suggests, this policy is meant for client systems primary. This policy outlines that IPSec security should only be used in response to a request by a server. As such, a client with this policy assigned will make all requests unsecured, but if a server requests security, is capable of responding using IPSec security.

Server (Request Security) - This policy, meant mainly for servers, will request that a client use a secured connection if assigned. The caveat here is that the server will only ask. If a client is not capable of using security (or has no policy configured), it will still allow unsecured communication. Think of this policy as provided best-effort IPSec security.

Server (Require Security) - Again a policy meant for servers, this one is a little harsh, and probably won't meet your needs. If configured, this server requires IPSec security from all connections, right from the get-go. Make an unsecured request and you won't be asked, informed, or otherwise - you'll simply be denied.

The three policies provided are simply a starting point, since you will likely need something a little more granular that meets your specific needs. To create a new IPSec policy, right-click and choose 'Create IP Security Policy'.

Doing so will open the IPSec Security Policy wizard, which will walk you through the policy creation process step-by-step. The first step is giving the policy a name, and an associated description. I would strongly recommend a name that will be descriptive enough to explain what the policy is used for.

The Request for Secure communication screen comes next. This allows you to control how this policy will respond when other servers on the network request secured communication.

Page 1 of 3


Comment and Contribute

Your name/nickname

Your email

(Maximum characters: 1200). You have characters left.