FreeBSD as a RAS for WIN2K and others IPSEC road warriors

Anton. R. Ivanov, IP Access LTD,
© 22.02.2002

 $Id: FreeBSD-WIN2K-IPSEC-HOWTO.html,v 2.16 2002/07/22 08:27:05 aivanov Exp $

1. Introduction and Disclaimers

FreeBSD on a common desktop class machine is sufficient as a RAS for the needs of a small enterprise. It can support up to 256 clients, more than 4 MBit 3des and more than 10MB des-cbc, blowfish or aes throughput on a 700MHz PIII. If you need more than this you most likely will have to go for a specialized access point from Cisco, Nortel or other vendors and/or specialized client software that supports algorithms more efficient than 3des.

These numbers are quoted for the specific deployment scenario when FreeBSD accepts connections from Windows2000 and XP machines that do not run any additional software. Without additional software these OSes can do only PPTP and L2TP VPNs. IPSEC is simply an additional protection. This results in the following peculiarities (I will intentionally avoid classifying them as advantages or disadvantages):

Initially I have designed the setup described in this document for Windows only. It currently supports Linux with FreeSWAN 1.96 after it has been patched with the x509 patch. It can also be extended further to support other VPN clients that can do PPTP or L2TP and IPSEC with certificates. I have added the clients other then windows for reasons of the "because it can be done" variety rather then because of a specific need. Most of these can do IPSEC tunnel mode and are better behaved in tunnel mode as well. There are not that many cases for this approach for non-windows systems. For example:


All usual disclaimers apply:

  • This worked for me, your mileage may vary.
  • My employer denies any responsibility for any damages or losses caused by the use or misuse of this information. I deny any reponsibility as well.
  • All trademarks are property of their respective owners.
  • This material is based on the excellent OpenSSL cookbook, OpenSSL documentation and various documents from Microsoft , Verisign, the KAME project and others. All information used in this HOWTO is either freely available on the Internet or can be derived from elementary interoperability testing.
  • I will be glad to accept any corrections and amendments to this document and maintain it.
  • You are allowed to redistribute this document free of charge, modify it as you see fit and include in other documents and products as long as the original copyright notice is retained and/or due credit is given.
  • 2. Certificate Authority.

    If you already have one - skip to section 2.3. Otherwise you will need to create a certificate root.

    2.1. Create the necessary directory structure

    Choose your own directory location. For example: /usr/local/etc/openssl. Create the necessary subdirectories and files:

    2.2. Prepare an openssl.conf file

    The only reason for doing this is to avoid having to enter your company name, location, etc every time you run openssl. Here is a sample config based on the SSL Cookbook. Notes:

    2.3. Create the CA certificate and make a copy of it in PKCS#12 format so Windows can read it.

    As a result you will have two PEM format files for your Certificate Root and a PKCS#12 file which you can import into a windows system. Keep in mind that if you do not use the -nokeys switch, the resulting PKCS#12 file will contain your CA private key as well. At least with OpenSSL 0.9.6c under debian-woody (which is what I use for a cert authority) this option does not work correctly and the keys are present in the PKCS#12 file (according to windows). If your private key has been kept for any reason it can be easily stripped by exporting it under windows and reimporting it back in.

    2.4. Create a cert request for the server, sign it and make a non-encrypted copy of it.


    2.5. Create a cert request for each windows machine, sign it and make windows readable PKCS#12 copy of it.

    Similar to the above: Notes:

    3. Setting up a FreeBSD box.

    3.1. Rebuild your kernel with IPSec.

    You will need to enable the following in the config file: Also enable the firewall of your choice IPFIREWALL or IPFILTER. Keep in mind that some options like NAT may not be compatible with using IPSEC.


    3.2. Build and install KAME racoon from the ports collection.

    It is under /usr/ports/security/racoon. In most cases a simple make install should be enough. Best of all put the config under its own directory like /usr/local/etc/racoon/. The config is quite simple: I have set it for the most elementary method of locally valid certificates (no certificate chain). When used this way racoon searches for certificates based on their checksum. So, all client and certification authority certificates under the cert directory will need to be symlinked to their magic hash names. For example: Notes:

    3.3. Build and install the PoPToP PPTP server.

    It is under /usr/ports/net/poptop. FreeBSD by default builds it with userland ppp which by default does not support RC4 encryption (it can be rebuilt with it). Considering that encryption is delegated to IPSec this is not a problem. Alternatively there is another VPN daemon and kernel ppp both of which support RC4 but I have not tested them. At least the kernel ppp with the RC4 patches is reported to be unstable. It may also be illegal to use it in some countties.

    After building it you need to add the following section to the bsd ppp options file.

    Update the IPs in the pptpd.conf file as well. I have not had the time to try what happens if they differ. If they are the same it works.


    3.4. Create a traffic policy

    Traffic policy on the BSD side is fairly simple. All GRE packets and PPTP control channel traffic is set to require IPSEC.

    4. Windows 2000 Setup

    You will need admin privileges for every client. There are also some prerequisites. Windows must be patched to SP2 and have the high encryption pack installed. The high encryption pack is available from Notes:

    4.1. Certificate Setup

    Note, that this is the part where it is easiest of all to make a mistake. From the start menu run "mmc". From the console menu chose add-remove snap-in. Add the certificate and the IPSEC snap-ins. When asked which certificates do you want to manage chose local computer. Note that windows has per-user certificates as well as per-computer ones, but the per-user ones are not useable for IPSEC.

    Under the Certificates, right button click on Trusted Root Certification authorities. From all actions choose import . Import your certificate authority PKCS#12 certificate here. Check if it is displayed.

    Under the Certificates, right button click on Personal. From all actions choose import. Import the user PKCS#12 certificate here. Check if it is correctly displayed and the root authority is known.

    Under IP Security policies choose a policy which you are not using or create a new one. Specify it as follows:

    If your personal certificate is correctly signed it will than be used for isakmp key negotiation. If you are having trouble, run racoon in debug mode on the BSD box. It will display the whole certificate in the debug output.

    4.2. PPTP setup.

    Create a new VPN connection. Make sure that under properties, in the security section the require data encryption field is left blank. You do not need it. This is the RC4 encryption for the PPTP which will be unnecessary overhead if IPSEC is protecting the connection. It may also be a good idea to force the connection to be a PPTP one. Windows 2000 tries L2TP first, so leaving this setting on automatic may slow down the connection process.

    5. Linux Setup

    I have used debian woody and compiled freeswan-1.96 from the unstable distribution. The reason for this choice is that the package in unstable already has x509 and several other important patches applied. If you are on a non-debian system I would suggest to start from the debian sources and patches in order to minimise the hassle. Otherwise the x509 patch is available from

    Overall, the linux IPSEC stack still has a way to go and is not 100% integrated into linux networking. Also, the syntax of ipsec.conf leaves more to be desired. There are several strong sides as well, like CRL support but they do not really come into play as a client.

    5.1. FreeSWAN installation

    Obtain freeswan if possible patched with the x509 patch and install it according to the instructions supplied on freeswan's web site and the instructions for the x509 patches.

    5.2. FreeSWAN configuration

    Extract the subject lines out of the certificates on the linux system and the FreeBSD VPN access machine. In theory you should not need to specify these in ipsec.conf. In practice I could not make it work. If you specify leftid=%cert or rightid=%cert for the system identity it will still supply/ask for an IP address instead of ASN1_DN in IKE phase 1.

    This results in the following config


    5.3. Networking.

    There is a pptp client package available nowdays in almost all distributions. Once again it has been tested on debian, so if there is no pptp package on your system I would suggest using debian sources.

    The setup from now on revolves around the problem of left=%defaultroute as well as several other problems related to the fact that IPSEC is still not properly integrated into linux networking code


    If you run into trouble your best chance is to use racoon in non-daemon mode with debugging enabled. In some cases having a look at the messages left by pptpd in the syslog may be necessary as well.

    I found most errors to be self-explanatory except one:

    This happens when the certificate used for the BSD server has an encrypted private key. You will need to run openssl rsa in order to decrypt it. This happens either when Windows 2000 has not been ServicePack2-ed or has remnants of some 3rd party tunnel mode software. Some of this software disables the Windows IKE infrastructure and uses its own DIY keying instead. You will need to reenable the service. There are two possible reasons for this. One is IKE not negotiating after a failure in modes different from aggressive. I have not debugged this one fully yet. In some cases only a reboot on the Windows machine may help. Move to aggressive mode if possible (see notes).

    Sequencing problems with the GRE packets. Windows VPN stops working if the packets are reordered along the way. Same goes for the pptpd implementation. This is a known problem and there is nothing you can do about it besides fixing the network or changing the provider. If the reordering occurs in your network look for any of the following:

    This problem will also occur if the windows network interface drops and comes back with the same IP address. Most common reason for this is network filters or IPSEC policy that prohibit DHCP on the Windows machine. These are visible in the windows log. Check and fix the policy accordingly. Note: If you wait for the normal redial interval the link will come back OK. Problem occurs only if you press the redial button too fast.

    These condition can be easily discovered on the pptpd side (large and small jumps in GRE sequence numbers) and I will hopefully fix them once I have some time to get to it. Watch this place ;-) This of course lives the windows side but not like there is anything I can do there.