PlayInfoBox

Home /Network /IPsec Explained: AH vs. ESP, Transport Mode vs. Tunnel Mode

IPsec Explained: AH vs. ESP, Transport Mode vs. Tunnel Mode

IPsec quietly powers most corporate VPNs behind the scenes β€” here's how AH, ESP, transport mode, tunnel mode, and IKE all fit together.

What IPsec is β€” a security toolkit that works at the IP layer

Unlike security technologies such as HTTPS that operate at the application (browser) level, IPsec (IP Security) works one layer down, at the IP layer itself. That means it can automatically encrypt and authenticate traffic from any application the moment it goes out as an IP packet, which is why it's the core technology behind most corporate site-to-site and remote-access VPNs.

AH (Authentication Header) β€” verifies integrity and origin, doesn't encrypt

AH (Authentication Header) attaches signature-like authentication data to a packet, letting the receiver confirm the packet wasn't tampered with in transit and really came from the claimed sender. It doesn't encrypt the actual content, though, so it can't prevent eavesdropping β€” which is why ESP, which adds encryption, is far more widely used in practice today.

ESP (Encapsulating Security Payload) β€” adds encryption, the standard in practice

ESP (Encapsulating Security Payload) encrypts the actual data content to prevent eavesdropping, while also providing integrity verification. Because it delivers both encryption and authentication, and handles firewall/NAT environments more gracefully than AH (AH signs the entire header, which tends to break when packets pass through NAT, while ESP is comparatively more flexible here), nearly all modern IPsec implementations use ESP as the default.

Transport Mode β€” protects only the payload, keeps the original IP header

In transport mode, the original IP header is left untouched, and only the actual data (payload) inside is encrypted and authenticated. It's mainly used for direct communication between two devices, has relatively low overhead, but exposes the real source and destination IP addresses of the two communicating devices.

Tunnel Mode β€” wraps the entire packet in a new IP header

Tunnel mode encrypts the entire original packet, header included, and wraps it with a brand-new IP header for delivery over a new path between gateways. As a result, the internal network's actual IP ranges stay hidden from outside view, functioning like a secure tunnel between two locations β€” this is the most common mode used for corporate site-to-site VPNs.

IKE β€” how both sides negotiate encryption keys safely

IKE (Internet Key Exchange) is the process by which two parties safely agree on which encryption method and key to use before starting IPsec communication. IKE uses techniques like Diffie-Hellman key exchange to generate a shared key securely even in the presence of an eavesdropper, then uses that key to establish the security association (SA) needed for actual AH/ESP communication. The improved IKEv2 is widely used today.

Real-world use β€” site-to-site VPNs and L2TP/IPsec

In corporate environments, IPsec (typically paired with IKEv2) is widely used for always-on site-to-site VPNs connecting branch offices to headquarters. For personal VPN services and remote-access VPNs, L2TP/IPsec β€” combining the L2TP tunneling protocol, which has no built-in encryption of its own, with IPsec β€” is common, with IPsec handling the actual encryption and authentication while L2TP handles building the tunnel itself.

Security Association (SA) β€” the agreed set of rules for one direction

A security association is the bundle of settings both sides agree on: the protocol (ESP or AH), the encryption and integrity algorithms, the keys, and a lifetime. Each SA is one-way, so a normal two-way tunnel uses a pair. Packets carry a Security Parameter Index (SPI) that tells the receiver which SA applies, and a sequence number that helps detect replayed packets. When the lifetime runs out, the peers renegotiate and replace the keys without interrupting traffic.

Protocol numbers and ports that firewalls care about

ESP is IP protocol number 50 and AH is protocol number 51, which are separate from TCP and UDP ports. IKE negotiation uses UDP port 500. When a device sits behind NAT, IKE and ESP traffic are usually wrapped in UDP on port 4500, a feature called NAT traversal. If an IPsec tunnel fails to come up, the first thing to check is whether a firewall or router in the middle blocks UDP 500, UDP 4500 or ESP.

Pre-shared keys versus certificates

Peers must prove who they are before building a tunnel. A pre-shared key is a secret string configured on both sides: it is simple but weak if the secret is short or reused across many sites. Certificates use public-key cryptography and a trusted authority, which scales better and avoids sharing a single secret, but they require more setup and management. Modern remote-access VPNs often combine certificates for the server with a username and password or another factor for the user.

Overhead, MTU, and why big packets can fail

IPsec adds headers and padding to each packet, and tunnel mode adds a whole new IP header. If the result is larger than the path's maximum transmission unit (MTU), the packet must be fragmented or dropped, which can cause strange symptoms such as small pages loading while large downloads hang. Administrators commonly lower the MTU or the TCP maximum segment size on the tunnel to prevent this. This kind of problem is a frequent cause of a tunnel that is up but seems unusable.

Working one layer below the application makes IPsec universal

Because IPsec operates at the IP layer rather than inside any single application, it protects traffic regardless of which app generated it β€” a browser, an email client, or a database connection all get the same protection automatically once IPsec is active on the connection. That's the core reason it became the default choice for securing entire network links rather than individual app sessions.

ESP and tunnel mode became the practical default combination

While AH and transport mode both exist as valid parts of the IPsec standard, real-world deployments have converged heavily on ESP (for encryption plus authentication) paired with tunnel mode (for full packet protection and hidden internal addressing) β€” largely because that combination handles modern network realities like NAT and firewalls far more smoothly than the alternatives.

IPsec compared with other VPN technologies

IPsec is standardized and widely supported by routers, firewalls and operating systems, which makes it a common choice for connecting sites from different vendors. SSL/TLS-based VPNs and OpenVPN work at a higher level and are often easier to pass through restrictive firewalls, while WireGuard is a newer protocol designed for a small code base and simple configuration. None of them is best in every case; the right pick depends on the equipment you have, who must connect, and how much configuration effort you accept.

Troubleshooting a tunnel that will not come up

Work through it in order. First confirm basic reachability between the two gateway addresses. Then check that both sides use the same settings for the IKE version, the encryption and hash algorithms, the Diffie-Hellman group, the authentication method and the lifetimes. Check that the local and remote subnets are mirror images of each other, because a mismatch in the traffic selectors is a very common cause of failure. Verify that firewalls pass UDP 500, UDP 4500 and ESP, and look at the logs on both ends, since the error on one side is often explained by the other side's message.

Frequently Asked Questions

Do I need to choose between AH and ESP myself when setting up a VPN?

Rarely β€” nearly all modern VPN software and hardware default to ESP because it covers both encryption and authentication, and most consumer or business VPN setups won't even present AH as a practical option.

What's the simplest way to remember the difference between transport and tunnel mode?

Transport mode protects the letter but keeps the original envelope showing; tunnel mode seals the entire original letter and envelope inside a brand-new envelope with a different address on it.

Why does IPsec often have problems behind home routers?

Home routers use NAT, which rewrites addresses and can confuse protocols that carry address information inside the packet. NAT traversal solves most of this by encapsulating ESP in UDP, but older devices or restrictive networks may still block it.

Is IKEv2 really better than IKEv1?

In most respects yes: IKEv2 is simpler, has fewer message exchanges, supports reconnection after a network change, and is better suited to mobile devices. Some older equipment supports only IKEv1, so check what both ends can use.

Does IPsec hide which websites I visit?

It encrypts the traffic between the two tunnel endpoints, so an observer between them cannot read the content or the inner destinations. The VPN gateway at the far end still sees the destinations, and the traffic is not private once it leaves the tunnel.