A VPN (Virtual Private Network) creates an encrypted tunnel between two points over an otherwise untrusted network (typically the public Internet), making traffic passing through that tunnel unreadable to anyone observing the underlying connection, and allowing a remote device to behave as though it’s directly connected to a private network.
There isn’t a single “VPN protocol” — several different protocols exist to build this tunnel, each with different trade-offs in security, performance, and complexity. This page covers the three most significant: IPSec, OpenVPN, and WireGuard.
What a VPN actually does
Two related but distinct things are usually happening at once when a VPN connection is established:
- Encapsulation: The original network packet is wrapped inside a new outer packet, allowing it to be carried across a network it wouldn’t otherwise be able to traverse directly (for example, letting a remote device appear to have an internal, private IP address even though it’s physically located elsewhere).
- Encryption: The contents of that wrapped packet are encrypted, so anyone intercepting the traffic between the two VPN endpoints only sees encrypted data, not the original packet or its contents.
Broadly, VPNs fall into two categories: site-to-site VPNs, which permanently connect two networks together (for example, linking a company’s head office and branch office), and remote-access VPNs, which let an individual device connect into a network temporarily, from wherever it happens to be.
IPSec (Internet Protocol Security)
IPSec is not a single protocol but a framework — a suite of related protocols that work together to provide encryption and authentication at the Network layer (Layer 3), meaning it can secure any traffic riding on top of IP, regardless of what higher-layer protocol is being used.
Key components
- IKE (Internet Key Exchange): Handles the negotiation phase — authenticating both ends of the connection and agreeing on a shared secret key, similar in spirit to a TLS handshake (see the TLS page).
- ESP (Encapsulating Security Payload): The component that actually encrypts the data and provides integrity checking. This is what most people mean when they refer to “the encryption” in an IPSec VPN.
- AH (Authentication Header): Provides integrity and authentication, but notably not encryption. AH is rarely used on its own in modern deployments, since ESP alone (or ESP combined with AH) already covers most practical needs.
Tunnel mode vs Transport mode
IPSec can operate in two modes:
- Tunnel mode: The entire original IP packet is encrypted and wrapped inside a new packet with new IP headers. This is the mode used for typical site-to-site and remote-access VPNs, since it fully hides the original source and destination addresses from anyone observing the outer traffic.
- Transport mode: Only the payload of the original packet is encrypted; the original IP headers remain visible. This is less common for full VPN use, and is more often seen protecting traffic directly between two specific hosts rather than routing an entire remote network’s traffic.
IPSec is a mature, widely supported, and thoroughly vetted standard — it’s built into most routers, firewalls, and operating systems, and remains extremely common in enterprise site-to-site VPNs. Its main drawback is complexity: IPSec has a large number of configurable options and parameters, and misconfiguration (particularly mismatched settings between two devices trying to establish a tunnel) is a very common source of practical connectivity problems.
OpenVPN
OpenVPN is an open-source VPN protocol that takes a different approach to IPSec — rather than operating directly at the Network layer, it runs as a regular application, using TLS (the same protocol behind HTTPS) to secure its connections, and can run over either TCP or UDP.
Because OpenVPN can be configured to run over TCP port 443 — the same port normally used for HTTPS traffic — its traffic can sometimes be harder to distinguish from ordinary web browsing on a network that’s otherwise restrictive about what traffic is permitted out. This has made it a popular choice in situations where a VPN needs to reliably work through firewalls or restrictive networks that might block other, more easily identifiable VPN protocols.
OpenVPN’s use of a full TLS handshake and its flexible configuration options make it well-tested and highly configurable, but generally slower to establish a connection and less efficient than the newer approach taken by WireGuard, below.
WireGuard
WireGuard is a significantly newer VPN protocol (first released around 2016, and merged into the Linux kernel in 2020), designed with a very different philosophy to both IPSec and OpenVPN: radical simplicity.
Where IPSec offers a large number of configurable cryptographic algorithms and options (a strength for flexibility, but also a source of complexity and potential misconfiguration), WireGuard deliberately supports only one, carefully chosen, modern set of cryptographic primitives, with no negotiation required. This design choice is a big part of why WireGuard’s entire codebase is dramatically smaller than IPSec’s or OpenVPN’s implementations — often cited as being around a few thousand lines of code, compared to hundreds of thousands for the alternatives — which makes it considerably easier to audit for security flaws.
How WireGuard works, briefly
Each WireGuard peer (client or server) has a simple public/private key pair, similar in concept to an SSH key pair (see the SSH page). A peer’s configuration simply lists the public keys of the other peers it’s willing to communicate with, along with which internal IP addresses and networks that peer is responsible for. There’s no separate authentication step involving usernames and passwords in the way IPSec or OpenVPN typically require — possession of the correct private key is what proves identity.
This simplicity also brings a significant performance benefit: WireGuard’s connection setup is extremely fast, and it’s generally considered to have lower overhead and better throughput than both IPSec and OpenVPN in most real-world comparisons, which is part of why it has been rapidly adopted by many commercial VPN providers and self-hosting tools since its release.
Comparing the three systems
| IPSec | OpenVPN | WireGuard | |
|---|---|---|---|
| Layer | Network (L3) | Application (uses TLS) | Network (L3) |
| Codebase size | Large | Large | Very small |
| Configuration flexibility | Very high | High | Deliberately minimal |
| Typical use case | Enterprise site-to-site | Restrictive networks, wide compatibility | Modern remote access, self-hosting |
| Performance | Good | Moderate | Excellent |
VPN protocols and security
VPNs are widely used as a security control in their own right — for example, the RDP hardening advice on this site recommends requiring a VPN connection before RDP is ever reachable at all (see the RDP page) — but the VPN itself needs to be properly configured to actually provide that protection.
Weak or outdated cryptographic settings are a real risk specifically with IPSec and OpenVPN, given how configurable both are — an IPSec tunnel configured to still permit an old, weak encryption algorithm for compatibility reasons provides a much weaker guarantee than the “IPSec” name alone might suggest. This is part of why WireGuard’s lack of configurability is sometimes considered a security feature rather than a limitation — there’s no weak option to accidentally select.
Split tunnelling is a common configuration choice with real security trade-offs. A VPN can be configured so that all of a client’s traffic routes through the tunnel (“full tunnel”), or so that only traffic destined for the private network goes through the tunnel while everything else (general web browsing) goes directly out the client’s own Internet connection (“split tunnelling”). Split tunnelling is often preferred for performance and bandwidth reasons, but it means a compromised client device retains a direct, un-monitored path to the wider Internet even while connected to the VPN, which can undermine security policies that assume all traffic is being inspected.
VPN endpoint compromise remains a real target. Because a VPN gateway is specifically the device responsible for handling encrypted traffic into a private network, VPN appliances themselves have been the subject of a number of serious, actively exploited vulnerabilities in commercial products over the years — a reminder that a VPN is only as secure as the software running the endpoint terminating it, not just the strength of the underlying protocol.
A VPN encrypts traffic — it doesn’t inherently authenticate what that traffic is used for. A common misconception is that connecting via VPN makes all subsequent activity implicitly safe. In reality, a VPN simply extends the private network’s boundary to include the remote device; if that device is itself compromised (for example, via malware), a VPN connection can just as easily give an attacker a path directly into the private network as it gives a legitimate user access.