Configuring and Managing IPSec

IPSec Review

IPSec is a framework of open standards for encrypting TCP/IP traffic within networking environments. IPSec works by encrypting the information contained in IP datagrams through encapsulating to provide data integrity, data confidentiality, data origin authentication, and replay protection.

IPSec uses cryptography to provide authentication, data integrity, and data confidentiality services. Authentication deals with verifying the identity of the computer sending the data, or the identity of the computer receiving the data. IPSec can use digital certificates, the Kerberos v5 authentication protocol, or pre-shared keys as an authentication method. Anti-replay ensures that the authentication data cannot be interpreted as it is sent over the network. IPSec can provide nonrepudiation. With nonrepudiation, the sender of the data cannot at a later stage deny actually sending the data. Data integrity deals with ensuring that the data received at the recipient has not been tampered with. A hashing algorithm is used to ensure that the data is not modified as it is passed over the network. The hashing algorithms which can be used by IPSec are Message Digest (MD5) ad Secure Hash Algorithm 1 (SHA1). Data confidentiality ensures that data is kept private by applying encryption algorithms to data before it is sent over the network. IPSec uses encryption algorithms such as Data Encryption Standard (DES), Triple DEC (3DES), or 40-bit DES to provide data confidentiality.

IPSec uses the Authentication Header (AH) protocol and Encapsulating Security Payload (ESP) protocol to provide data security on client computers, domain servers, corporate workgroups, LANs, WANs and remote offices. The Authentication Header (AH) protocol provides data authentication and integrity, and can therefore be used on its own when data integrity and authentication are important to the organization but confidentiality is not. The AH protocol does not provide for encryption, and therefore cannot provide data confidentiality. The Encapsulating Security Payload (ESP) protocol ensures data confidentiality through encryption, data integrity, data authentication, and other features that support optional anti-replay services. To ensure data confidentiality, a number of encryption algorithms are used. The main difference between the AH protocol and the ESP protocol is that the ESP protocol provides all the security services provided by the AH protocol, together with data confidentiality through encryption.

When you install IPSec, the two main IPSec components which are installed are the IPSec Policy Agent and the IPSec driver. The IPSec Policy Agent is a service running on a Windows Server 2003 computer that accesses IPSec policy information. The IPSec Policy Agent accesses the IPSec policy information in the local Windows registry or in Active Directory. The IPSec Policy Agent then passes information to the IPSec driver. The IPSec driver performs a number of operations to enable secure network communications such as initiating IKE communication, creating IPSec packets, encrypts data, and calculates hashes.

IPSec can operate in either Tunnel mode or in Transport mode. IPSec Tunnel mode should be used to provide security for WAN and VPN connections that use the Internet. In tunnel mode, IPSec encrypts the IP header and the IP payload. With tunneling, the data contained in a packet is encapsulated inside an additional packet. The new packet is then sent over the network. In Transport Mode, the default mode of operation used by IPSec, only the IP payload is encrypted. Transport mode is used for end-to-end communication security between two computers on the network. 

IPSec policies are used to apply security in your network. The IPSec policies define when and how data should be secured. The IPSec policies also determine which security methods to use when securing data at the different levels in your network. You can configure IPSec policies so that different types of traffic are affected by each individual policy. IPSec policies can be applied at the Active Directory domain level, site level, OU level, and it can be applied on computers and applications. You can use a security method to define how an IPSec policy should deal with traffic matching an IP filter.A filter action can be used to define how the IPSec driver should secure traffic. The filter actions result in either of the following events: Drops traffic, Allows Traffic, or Negotiates security. IPSec filters can be used to filter out unauthorized traffic. Through security filters, you can filter traffic according to the following: Traffic allowed, Traffic to secure, or Traffic to block. You can also group multiple security filters into a filter list.

IPSec Management Tools

The IP Security Policy Management snap-in is used to manage IPSec, create IPSec policies and edit existing IPSec policies, and assign IPSec policies. You can also use the IP Security Policy Management tool to add and remove filters which are applied to IPSec policies. To work with the IP Security Policy Management snap-in, you first have to create a MMC console for the snap-in.

When you install the IPSec snap-in, you need to select which IPSec policy you want to manage, and on what network level you want to manage IPSec. You can select either of the following options:

You can also configure IPSec using the Netsh command-line utility. The netsh command-line utility replaces the previously used Ipsecpol.exe command-line utility. The netsh command-line utility cannot though be used to manage IPSec on Windows XP machines.

The Netsh command-line utility can be used to manage the following aspects of IPSec:

Windows Server 2003 includes the new IP Security Monitor tool which is implemented as an MMC snap-in. The IP Security Monitor tool provides enhanced IPSec security monitoring.

With the IP Security Monitor tool, you can perform the following administrative activities:

IPSec supports the new Resultant Set of Policy (RSoP) feature of Windows Server 2003. The Resultant Set of Policies (RSoP) calculator can be used to determine the policies which have been applied to a particular user or computer. Resultant Set of Policy (RSoP) sums all group policies which are applied to a user and computer in a domain. This includes all filters and exceptions. You can use the feature through the Resultant Set Of Policy (RSoP) Wizard or from the command-line to view the IPSec policy that is applied.

How to create an MMC console for the IP Security IP Security Policy Management snap-in

  1. Click Start, click Run, type mmc in the Run dialog box, and then click OK.
  2. Click the File Menu item, and select Add/Remove Snap-in.
  3. The Add/Remove Snap-in dialog box opens. Click Add.
  4. The Add Standalone Snap-In dialog box opens.
  5. In the Available Standalone Snap-ins list, select IP Security Policy Management, and then click Add.
  6. The Select Computer Or Domain dialog box opens.
  7. Click the Local Computer option.
  8. Click Finish.
  9. Click Close to close the Add Standalone Snap-in dialog box.
  10. Click OK to close the Add/Remove Snap-in dialog box.

How to enable IPSec for an Active Directory domain

  1. Click Start, click Run, type mmc in the Run dialog box, and then click OK.
  2. Click the File Menu item, and select Add/Remove Snap-in.
  3. The Add/Remove Snap-in dialog box opens. Click Add.
  4. The Add Standalone Snap-In dialog box opens.
  5. Select Group Policy Object Editor, and then click Add.
  6. The Select Group Policy Object dialog box opens. Click Browse
  7. The Browse For A Group Policy Object dialog box opens.
  8. Select Default Domain Policy, and then click OK.
  9. Click Finish.
  10. Click Close to close the Add Standalone Snap-in dialog box.
  11. Click OK to close the Add/Remove Snap-in dialog box.
  12. Expand Domain Policy, expand Computer Configuration, expand Windows Settings, expand Security Settings, expand IP Security Policies on Active Directory.
  13. Select IP Security Policies On Active Directory.
  14. The details pane displays all available IPSec policies.
  15. Right-click the IPSec policy which you want to assign, and then click Assign from the shortcut menu.

How to use the netsh command-line utility to manage IPSec

Before you can use the netsh command-line utility to manage IPSec, you first have to change the utility to the ipsec context.

The two types of command modes which you can utilize for IPSec makes different IPSec functions available:

To change the netsh command-line utility to the ipsec context,

  1. Click Start, click Run, type cmd, and then click OK.
  2. When the command prompt window, enter netsh ipsec.

A few typically used netsh commands and switches are:

Configuring IPSec

You configure IPSec through configuring the following aspects of IPSec policies:

You can use either of these methods to configure IPSec policies:

When you open the Security Policy Management snap-in, the following predefined default IPSec policies are displayed:

You can also create customized IPSec policies that include customized rules and filters that suit specific security requirements of the organization. Customized IPSec policies can be created in the IP Security Policy Management MMC. You can also create your own IPSec policy by using the IP Security Wizard which you can initiate from within the IP Security Policy Management MMC.

After you have followed the prompts of the IP Security Policy Wizard to create a create your own IPSec policy, you can access the properties of the IPSec policy to add, remove, and configure rules, filters, filter lists, and security actions for the policy.

The Properties dialog box of an IPSec policy contains the following tabs:

If you have deselected the Use Add Wizard checkbox on the Rules tab of the Properties dialog box of an IPSec policy, and you click the Edit button on the Rules tab, then the Edit Rule Properties dialog box opens.

The Edit Rule Properties dialog box contains the following tabs which you can use to set configuration settings:

How to create a customized IPSec policy

  1. Open the IP Security Policy Management console.
  2. Select IP Security Policies.
  3. Right-click the security policy which you want to customize and then click Properties from the shortcut menu.
  4. When the Properties dialog box of the security policy opens, click the Rules tab.
  5. Find the rule which you want to change, and then click the Edit button.
  6. Click the Filter Action tab.
  7. Proceed to customize the filter action.
  8. Click the Security Methods tab.
  9. If you want to create a new security method, click Add.
  10. If you want to remove an existing security method, click Remove.
  11. If you want to change an existing security method, click Edit.
  12. To add a security method, click Custom, and then click Settings.
  13. The Custom Security Method Settings dialog box opens.
  14. To provide data integrity for the IP header and payload, enable the Data and address integrity without encryption (AH) checkbox. Select either the MD5 option or the SHA1 option as the Integrity algorithm.
  15. To provide data integrity and data confidentiality for the IP header and payload, enable the Data integrity and encryption (ESP) checkbox. Select either the MD5 option or the SHA1 option as the Integrity algorithm if you do not have the AH authentication protocol enabled. If you do have the authentication protocol enabled, click the None option. Select either the None option, DES option, or the 3DES option as the Encryption algorithm.
  16. To specify how many kilobytes of data must be transmitted before a new key is generated, click the Generate a new key every checkbox. Set a value that defines the number of kilobytes.
  17. To specify how many seconds must pass before a new key is generated, click the Generate a new key every option and enter a value which defines the number of seconds.
  18. Click OK.

How to create a new IPSec policy

  1. Open the IP Security Policy Management console.
  2. Right-click IP Security Policies and then select Create IP Security Policy from the shortcut menu.
  3. The IP Security Policy Wizard initiates.
  4. Click Next on the IP Security Policy Wizard Welcome page.
  5. On the IP Security Policy Name page, provide a name and a description for the new IPSec policy, and then click Next.
  6. On the Requests for Secure Communication page, you can leave the Activate the default response rule option selected, or you can deselect the option. Click Next.
  7. On the Default Rule Authentication Method page, set the authentication method for the security rule, and then click Next.
  8. On the Completing the IP Security Policy Wizard page, select the Edit properties option, and then click Finish.
  9. The IP Security Policy Properties dialog box for the new policy opens so that you can change the properties of the policy, and change any security rules.
  10. Click Edit on the IP Security Policy Properties dialog box.
  11. The Edit Rule Properties dialog box opens. The tasks which you can perform are listed here:
    • Add and remove security methods.
    • Modify existing security methods.
    • Set the order of precedence for security methods.
    • Specify the utilization of session key perfect forward secrecy (PFS).
  1. Click the Authentication tab. The tasks which you can perform are listed here:
    • Add and remove authentication methods.
    • Set the order of precedence for authentication methods.
  1. Click OK to close the Edit Rule Properties dialog box.
  2. Before you assign the IPSec policy, first ensure that the IPSec service is running.
  3. In the IP Security Policy Management console, right-click the new policy name that you want to assign, and then click Assign from the shortcut menu.

Managing Filter Lists and Filter Actions

IP filters, IP filter lists, and filter actions and rules define the IP security policy. Filters define the source, destination and action for each specific rule.

When defining a filter action, you can choose between the following filter actions in the IP Security Policy Management MMC snap-in:

Before managing IPSec rules, consider the following factors:

You can manage filter lists and filter actions using either of these methods:

The information that you need to provide when you add a new filter or change an existing filter are listed here:

The IP Filter Properties dialog box can be used to modify a filter. When you click the Add or Edit buttons, the Properties dialog box of the specific filter opens, displaying the following tabs:

Click the Source Address drop-down box to set the source address:

To configure mirrored settings, select or deselect the Mirrored checkbox:

  • If you only want to create one filter, you should deselect or uncheck the Mirrored checkbox.
  • If you want to create two filters using the same filter settings, click the Mirrored checkbox. A filter will be created to filter traffic to the specific destination, and another filter will be created to filter traffic from the specific destination.
  • If you want to create a filter for an IPSec tunnel, then you need to create two filter lists and ensure that the Mirrored checkbox is cleared or unchecked.

How to modify an IP filter

  1. Open the IP Security Policy Management console
  2. Double-click the IPSec policy that you want to configure.
  3. Click the Rules tab.
  4. Double-click the IPSec rule which holds the filter that you want to change.
  5. Click the IP Filter List tab.
  6. Click the Add button to configure an additional filter list.
  7. Click the Remove button to remove an existing filter list.
  8. Click an existing filter in IP Filter Lists list, and then click the Edit button to modify it.
  9. Click OK.

Assigning IPSec Policies in Group Policy

Before configuring IPSec policies for domains for Active Directory, consider the following factors:

Best Practices for Implementing IPSec

A few recommendations for implementing IPSec, as defined by Microsoft, are listed here:



Top 5 Free Networking Tools

Bookmark Configuring and Managing IPSec

Latest Blog Posts


English English GermanGerman SpanishSpanish FrenchFrench ItalianItalian PortuguesePortuguese RussianRussian DutchDutch
GreekGreek HindiHindi JapaneseJapanese KoreanKorean ChineseChinese Chinese (Simplified)Chinese (Simplified) ArabicArabic

Copyright 2009 Tech-FAQ. All rights reserved. Privacy Policy.