IPsec is a framework of techniques used to secure the connection between two points. It stands for Internet Protocol Security and is most frequently seen in virtual private networks (VPNs). It can be somewhat complex, but it is a useful option for securing connections in certain situations.
This guide breaks IPsec down into easy chunks, giving you an introduction that covers what the protocol is, how it works, and some of its potential security issues.
IPsec: An Overview
IPsec was initially developed because the most common internet protocol, IPv4, doesn’t have a lot of security provisions in place. Data transmitted over IPv4 can easily be intercepted, altered or stopped, which makes it a poor system for any important transmissions.
A new set of standards was needed to protect information. IPsec filled this gap by acting as a framework that can authenticate connections, as well as prove the integrity of data and make it confidential. IPsec is an open standard that acts at the network level. It can be used to securely transfer data from host-to-host, network-to-network, or between a network and a host.
IPsec is most commonly used to secure traffic that passes over IPv4. Initially, there was also a requirement for implementations of the newer internet protocol, IPv6, to support IPsec. Despite this, it is now only a recommendation and is not enforced.
As a framework, IPsec it is made up of three main elements. The first two are the protocols, Encapsulating Security Payload (ESP) and Authentication Header (AH). Security Associations (SAs) are the final aspect.
ESP can be used to both encrypt and authenticate data, while AH can only be used to authenticate it. The two options are normally used separately, although it is possible to use them together.
IPsec uses SAs to establish the parameters of connections. These parameters include the key management systems that each party will use to authenticate each other, as well as encryption algorithms, hashing algorithms and other elements that are important for operating a secure and stable connection.
IPsec can use both ESP and AH in either tunnel or transport mode. When tunnel mode is used, the entire data packet is either encrypted or authenticated (or both). The payload, header and trailer (if included) are wrapped up in another data packet to protect it.
In transport mode, the original header remains, but a new header is added underneath. There are also some other changes, depending on whether ESP or AH is being used. This serves to protect the packet, however, some information is still available to attackers.
The most common configuration that we see is ESP with authentication in tunnel mode. This is what many VPNs rely on to secure data. It functions like an encrypted tunnel, giving data a safe passage as it passes through potentially dangerous intermediate networks.
The history of IPsec
In the early days of the internet, security wasn’t much of a priority in many situations. This is because the internet community was restricted to those who had the knowledge, resources, and desire to use it. The number of users was tiny in comparison to the modern day, and a much smaller amount of data was being transmitted. Because of this, attackers had far fewer opportunities.
As the community grew and the internet became more active, security became more of a necessity. In the eighties, the NSA used its Secure Data Network Systems (SDNS) program to fund the development of a number of security-focused protocols.
The results were published by the National Institute of Standards and Technology (NIST) in 1988. One of the protocols outlined by NIST, Security Protocol at Layer 3 (SP3), ended up becoming an Internet Standard, the Network Layer Security Protocol (NLSP).
Over time, various organizations began to improve upon SP3, with projects undertaken by the US Naval Research Lab (NRL), AT&T Bell Labs, Trusted Information Systems and others. At the same time, other advances, such as the device driver for the Data Encryption Standard (DES), made it possible to securely send data between the coasts of the US at reasonable speeds for the time period.
If these developments were left to continue separately, it would have led to interoperability issues between the various systems. The Internet Engineering Task Force (IETF) formed the IP Security Working Group to standardize these developments into an interoperable protocol. In 1995, the IETF published the details of the IPsec standard in RFC 1825, RFC 1826 and RFC 1827.
The NRL was the first to come up with a working implementation of the standard, which has since gone into mainstream use. Over the years, there have been numerous updates to IPsec and its documentation, but it is still deployed to authenticate both sides of a connection and secure the data that travels in between.
How does IPsec work?
Before we get into the more technical details of IPsec and its different modes, we will talk about it through an analogy that makes it easier to visualize the somewhat complicated configurations. First, you need to understand a bit about how packets work over IPv4, and the security problems associated with them.
What are data packets and how do they work?
Data is transmitted in packets, which consist of a payload and a header in IPv4. The payload is the data itself that is being transmitted, while the packet headers includes the type of protocol, addresses and other information needed to make sure that the data can arrive at its desired location.
One of the best ways to picture data packets is to think of them as postcards. The payload is the message that someone writes on the back, and the header is the delivery information that you put on the front. Just like with postcards, the data sent in a normal IPv4 packet is not very secure.
In these standard packets, anyone can see the payload, just like the postman or any attacker that intercepts your postcard can read it. These packets can even be altered as if you had initially written the postcard in pencil and an attacker erased the original message and wrote in something else.
Attackers can also see the header information, just like the front side of your postcard. This lets them know who you are communicating with and how you are doing it. They can even fake their own IPv4 packets to make them look like you sent them, which is a lot like if they copied your handwriting style and pretended to be you.
As you can see, this is hardly a secure method of communication, and there are many ways that it can be disrupted by attackers. This is what led to the development of IPsec. It provided a way to authenticate connections, prove the integrity of data, and keep it confidential, all at the network level. Without IPsec or other security protocols in place, attackers would be free to view or alter any sensitive and valuable data that they intercepted.
Encapsulating Security Payload (ESP): A visualization
We will talk about ESP first because this is the more commonly used protocol. When it is implemented with authentication in tunnel mode, it is used to form VPNs that safely connect hosts and networks across the insecure intermediate networks that lie in between.
As you saw above, it’s unsafe to transmit sensitive data with IPv4. IPsec’s ESP mode solves this issue by providing a way to encrypt the data, which makes the data confidential and prevents attackers from being able to access it. ESP can also be used to authenticate data, which can prove its legitimacy.
When ESP is used with encryption, it’s a lot like putting the postcard in a locked box and sending it via courier. No one can see the contents of the postcard, nor can they alter them. They can see whatever mailing information is written on the locked box, but this is different from what is written on the postcard.
If ESP is used with authentication as well, it’s a lot like signing the postcard or putting your own personal seal on it before it is placed in the box. When the recipient receives the locked box, they can open it up and take out the postcard. When they see the seal or the signature, then they know that the postcard is legitimately from you.
When ESP is in transport mode, it’s as though the postcard is locked in a box and sent by courier, except the box has a clear window through which you can see the address information of the postcard. When it is in tunnel mode, it’s like the postcard is in a solid box, with different address information on the outside.
Of course, this is all just an analogy to help you visualize what is going on. In reality, there are some pretty significant differences such as data traveling between networks and hosts, rather than just one person sending information to another.
Encapsulating Security Payload (ESP): The technical details
Now that we’ve given you a rough idea of how ESP works to protect data, it’s time to look at it on a more technical level. ESP can be used with a range of different encryption algorithms, with AES being one of the most popular. It can even be implemented without encryption, though this is rarely done in practice. The headers of ESP data packets have a Security Parameters Index (SPI) and a sequence number.
The SPI is an identifier which lets the recipient know which connection the data relates to, as well as the parameters of that connection. The sequence number is another identifier which helps to prevent attackers from altering data packets.
ESP also features a trailer, which contains padding, details on the protocol type for the next header, and authentication data (if authentication is being used). When authentication is in place, it is done with a Hashed Message Authentication Code (HMAC), which is calculated using algorithms like SHA-2 and SHA-3.
ESP authentication only validates the ESP header and the encrypted payload, but it doesn’t affect the rest of the packet. When a packet is encrypted with ESP, attackers can only see the data from the header and not the payload.
Encapsulating security payload (ESP): Transport mode
ESP’s Transport Mode is used to protect the information sent between two hosts. The IP header is kept on top of the ESP packet, and much of the header information remains the same, including the source and destination addresses. It encrypts and optionally authenticates the packet, which provides confidentiality and can be used to verify the packet’s integrity as well.
Encapsulating security payload (ESP): Tunnel mode
When ESP is in tunnel mode, the entire IP data packet is wrapped inside another one, and a new header is added on top. When authentication is also in place, ESP tunnel mode can be used as a VPN. This is IPsec’s most frequently used configuration.
The authentication, proof of integrity and confidentiality measures involved in ESP’s authenticated tunnel mode make it useful for safely joining two separate networks across the untrusted and potentially dangerous networks that lie between them.
This mode is best described by the common cliche of building an encrypted tunnel between the two points, creating a safe connection that attackers cannot penetrate. When ESP is used to encrypt and authenticate, attackers that intercept the data can only see that a VPN is being used for the connection. They cannot see the payload or the original header.
VPNs were initially used by businesses to connect regional offices with headquarters. This type of connection allows companies to easily and securely share data between their separate locations. In recent years, VPNs have also become a popular service for individuals. They are often used for geo-spoofing or for securing connections, particularly when using public wi-fi.
Authentication Header (AH): A visualization
Now that we have covered ESP, AH should be a little easier to understand. Since AH can only be used to authenticate data packets, it’s just like signing a postcard or adding your own seal to it. Because no encryption is involved, there is no locked box or courier in this analogy.
The lack of encrypted protection is a bit like a postcard that is sent through the regular mail, where the postman and any attackers can see both the address information, as well as the message that is written on the back of the card. However, due to the seal or signature, this information cannot be altered. Likewise, the seal and signature allow you to prove that you were the true sender, rather than someone that was trying to imitate you.
In AH transport mode, an authentication header is added, which protects the payload and the vast majority of the header information. This is kind of like an imaginary postcard that has a clear seal protecting the majority of its contents.
Anything underneath the seal cannot be changed, but the entire postcard can be seen by anyone who intercepts it. A few details aren’t covered by the seal and could potentially be altered, but all of the important information is secured by the seal. The seal would also have some information written on it, but for the purposes of this example, it’s not important to elaborate on it right now.
In tunnel mode, the packet is wrapped up inside of another one, and the majority is authenticated. The analogy breaks down a bit in this case, but it’s kind of like wrapping the postcard in a clear envelope with a seal. The envelope contains its own address information as well, and the seal protects almost the whole thing from being modified.
Authentication Header (AH): Technical details
AH is used to authenticate that data comes from a legitimate source, as well as that it retains its integrity. It can be used to show that data hasn’t been tampered with and to protect against replay attacks.
AH isn’t implemented very frequently, but it’s still important to discuss. It adds its own authentication header and uses Hash Message Authentication Codes (HMACs) to protect the majority of the data packet. This includes the entire payload, as well most of the fields in the header. The HMACs are calculated with hash algorithms like SHA-2.
Authentication Header (AH): Transport mode
AH transport mode is generally used for two-way communication between hosts. An AH header is added to the packet, and some of the protocol code is moved around. When an AH packet arrives and the HMAC is checked, the AH header is taken away and a few other changes are made. Once the data packet is back in its normal form, it can be processed as usual.
Authentication Header (AH): Tunnel mode
In this mode, the original packet is wrapped up inside another, then authenticated with an HMAC. This process adds an authentication header, authenticating the entirety of the original header (in transport mode, a few parts of the header are not covered) as well as the majority of the newly added header. The payload is also authenticated.
When these packets reach their destination, they undergo an authentication check, then the packet is brought back to normal by taking away both the newly added header, as well as the authentication header.
Security Associations (SAs)
Security Associations (SAs) set and store the parameters for an IPsec connection. They are used by both AH and ESP to establish a stable communication process that meets the security needs of each side. Each host or network has separate SAs for every party that it connects with, all of which have their own set of parameters.
When two hosts negotiate their connection for the first time, they form an SA with the parameters that will be used in the connection. They do this in a step-by-step process, with one party offering a policy that the other can either accept or reject. This process continues until they come up with a policy that is mutually agreeable and is repeated for each separate parameter.
Each SA includes the authentication and cryptographic algorithms that will be used. The SAs also include the parameters for key exchange (such as IKE), the IP filtering policy, routing restrictions and more. Once a Security Association has been established, it is stored in the Security Association Database (SAD).
When an interface receives a data packet, it uses three different pieces of information to find the correct SA. The first is the Partner IP address, which as you might assume, is the IP address of the other party in the connection. The second is the IPsec protocol, either ESP or AH.
The final piece of information is the Security Parameters Index (SPI), which is an identifier that is added to the header. It is used to choose between the SAs of different connections in order to make sure that the correct parameters are applied.
Each SA only goes one way, so at least two are needed so that communication can go in both directions. If AH and ESP were to be used together, then an SA will be needed in each direction for each protocol, totaling four.
IPsec vs. TLS/SSL
Sometimes it can be hard to understand the difference between IPsec and protocols like TLS/SSL. After all, they both just provide security, right? They do, but they do it in different ways and at different levels.
One of the best ways to compare IPsec and TLS/SSL is to look at them in the context of the OSI model. The OSI model is a conceptual system that is used to help understand and standardize the different aspects and layers of our complicated communication process.
In this model, IPsec functions at the third layer, the network layer, which means that it is positioned to transfer data packets to a host over a single or series of networks.
When we look at TLS/SSL, things get a little bit more confusing. This is because it runs over another transport medium, TCP. This should place TLS/SSL above layer four in the OSI model.
TLS/SSL also organizes handshake messages, which are the fifth level, the session layer. This would put TLS/SSL in either layers six or seven.
It gets more complicated when you consider that applications use TLS/SSL as a transport protocol. This would put TLS/SSL at level four or below. But how can it be simultaneously in layers six or seven, as well as in layer four or below?
The answer is that the TLS/SSL just doesn’t fit into the model. The reasons behind this are out of the scope of this article. The most important thing you need to know is that the OSI model is just that, a model, and sometimes reality doesn’t conform to our neat and tidy models.
When we talk about these standards in a practical sense, TLS/SSL’s role is to authenticate data, check its integrity, encrypt it, and compress it. It can be implemented to secure a range of other protocols, such as HTTP or SMTP, and is also seen in a wide range of applications, such as VoIP and web browsing.
IPsec differs in a couple of ways, the first is that it’s a framework, rather than a single protocol. It is also more complex, which makes it difficult to set up and maintain.
In the end, TLS/SSL is simpler than IPsec, which is another reason why it tends to be implemented in a more widespread manner. It is a core part of one of the main alternative tunneling protocols, OpenVPN.
See also: Ultimate Guide to TCP/IP
IPsec Security
There has been a lot of talk about government agencies placing backdoors into IPsec and taking advantage of vulnerabilities to target those using the protocol. Some of the earliest allegations came out in 2010, when Greg Perry contacted the lead developer of OpenBSD.
He asserted that the FBI had placed numerous backdoors, as well as mechanisms for side channel key leaking, into the OpenBSD code. This was alleged to affect the OpenBSD IPsec stack, which is widely used.
In 2013’s Snowden leaks, it was revealed that the NSA was targeting various forms of encryption and other security tools. The documents seem to confirm that the NSA had its own methods for accessing the keys used in IPsec, allowing them to snoop on certain connections.
NSA documentation leaked by the Shadow Brokers in 2016 showed that the agency has a tool which could be used to break the IPsec implementation used in Cisco’s PIX firewalls. Although these firewalls were discontinued in 2009, the BENIGNCERTAIN attack could be used to access the passwords for PIX devices.
The attack involved sending Internet Key Exchange (IKE) packets to a PIX server, which would cause it to release some of its memory. This information could be searched through to find the configuration information and the RSA private key, which could then be used by the NSA to spy on the IPsec connection. Bear in mind that this is just one implementation that was vulnerable, and it does not affect any current forms of IPsec.
In 2018, researchers exploited a flaw in the IKE protocol, which allowed them to decrypt connections. The proof-of-concept can be used to conduct man-in-the-middle attacks, where attackers can intercept data, tamper with it or even stop it from being transmitted.
The technique uses Bleichenbacher oracles to decrypt nonces, which disrupts the RSA authentication in phase one of IKE. This allows attackers to use fraudulently authenticated symmetric keys with their target. They can then spoof the IPsec endpoint, disrupting the encryption, which allows them to insert themselves into what was previously a secure connection.
While this is a worrying attack, patches have been released for the implementations that it is known to affect. Huawei, Cisco, Clavister and XyXEL all released patches soon after they were alerted about the vulnerability. It is safe to use these previously affected implementations as long as they are up-to-date.
There are several more potential vulnerabilities in IPsec, many of which involve IKE. Despite these issues, IPsec is still considered safe for general use, as long as it has been implemented properly and the implementation is using the latest updates.
IPsec itself is not broken. It’s important to remember that it’s just a framework, which can use a number of different protocols. It is still possible to use IPsec safely, as long as the right protocols are used in the appropriate manner. However, unsafe configurations have the potential to be attacked by the NSA and other parties, so it is important that you use IPsec correctly.
Should you use IPsec?
IPsec is mostly used to form a secure tunnel in VPNs, but it’s not the only option. If you are worried about your security, it’s best to consider the other options and find a solution which suits your use case and risk profile.
The main alternatives are PPTP, SSTP, OpenVPN, and WireGuard. The Point-to-Point Tunneling Protocol (PPTP) is old and has a lot of security issues, so it is best to avoid it in all circumstances. The Secure Socket Tunneling Protocol (SSTP) is a better option for Windows users, however, it is not independently audited.
OpenVPN is an open-source alternative that has a range of different configuration options. It uses SSL/TLS and is not known to have any security issues, so it’s a good choice for anyone who is concerned about the security problems that have been found in IPsec.
WireGuard is a lightweight open-source protocol that’s been around for a few years now. It uses ChaCha20 authenticated encryption by default, which has a shorter key than AES-256 encryption. This, together with its minimal code, has made WireGuard the go-to protocol for VPN services that prioritise speed.
The above doesn’t mean that all IPsec connections are inherently insecure, it just means there are safer alternatives for those who are security-conscious, or face a high threat-level.
Related: IPSec vs SSL
Computer keyboard licensed under CC0