What are Encryption Protocols and How Do They Work?

Published: May 2021. Updated: August 2026. Reviewed by Encryption Consulting’s PKI Advisory team.

Encryption turns readable data into ciphertext, but encryption alone does not tell two computers how to agree on a key, verify each other’s identity, or structure the encrypted bytes on the wire. That is the job of an encryption protocol. Every secure connection you rely on, a browser session, a VPN tunnel, an SSH login, a signed email, is built on a named protocol that specifies exactly how encryption gets used, not just which algorithm does the math. Picking the wrong one, or an outdated version of the right one, is one of the most common gaps our Encryption Advisory engagements find during architecture reviews.

What Is the Difference Between an Encryption Protocol and an Encryption Algorithm?

An encryption algorithm is a specific mathematical procedure, such as AES, RSA, or ECDSA, that transforms plaintext into ciphertext, generates a digital signature, or produces a hash. An algorithm on its own has no opinion about how two parties find each other, agree on which algorithm to use, exchange keys safely, or detect a tampered message.

A protocol is the rulebook that wraps one or more algorithms into a working security system for a specific purpose. TLS 1.3, defined in RFC 8446, specifies the exact handshake messages a client and server exchange, which cipher suites are permitted, how the session key is derived, and how the connection is authenticated with a certificate. Swap AES for ChaCha20-Poly1305 inside that same TLS 1.3 handshake and the protocol has not changed, only the algorithm the protocol negotiated. That distinction matters operationally: a protocol version can be deprecated even when every algorithm inside it is still considered strong, because the negotiation logic itself has a flaw (this is exactly what happened to SSLv3, discussed below).

Working alongside most asymmetric protocols is Public Key Infrastructure (PKI). A PKI issues the digital certificates that TLS, IPsec, and S/MIME use to bind a public key to an identity, so the protocol’s handshake has something trustworthy to check against a chain of trust back to a Certificate Authority. Without that identity layer, a protocol can still encrypt a channel, but it cannot tell you who is on the other end of it.

What Are the Major Encryption Protocols and How Does Each One Work?

TLS/SSL: Securing Client-Server Connections

Transport Layer Security (TLS), the successor to the older Secure Sockets Layer (SSL), is the protocol behind the padlock icon and “https” in your browser’s address bar. TLS does not encrypt anything itself; it negotiates which algorithms will do that work. A TLS handshake selects the protocol version, authenticates the server (and optionally the client) using an X.509 certificate, agrees on a cipher suite, and derives a shared session key. TLS 1.3, standardized in RFC 8446, cut the handshake to one round trip, removed support for static RSA key exchange and CBC-mode ciphers, and restricts negotiation to a short list of AEAD cipher suites. See our introduction to cipher suites for exactly which algorithm combinations TLS 1.2 and 1.3 permit and how the handshake negotiates them.

IPsec: Encrypting Network Traffic and VPN Tunnels

Internet Protocol Security (IPsec) secures traffic at the network layer instead of the application layer, which makes it protocol-agnostic: anything riding on IP gets protected without the application needing to know encryption is happening. IPsec has two core sub-protocols, AH (Authentication Header) for integrity and ESP (Encapsulating Security Payload) for confidentiality plus integrity, and two modes: transport mode, which encrypts only the packet payload, and tunnel mode, which encrypts the entire original packet including its header. Tunnel mode is what most site-to-site and remote-access VPNs use. Before any of that encryption can happen, both endpoints need a shared key, which is where the Internet Key Exchange (IKE) protocol comes in. IKEv2, defined in RFC 7296, negotiates and refreshes those keys and is the current standard; its predecessor IKEv1 was formally deprecated by RFC 9395 in 2023 and should be retired from any configuration still running it.

SSH: Securing Remote Administration

Secure Shell (SSH) protects interactive remote logins, file transfers, and port forwarding. Its architecture, described in RFC 4251, is split into three layers, each with its own RFC: the transport layer (RFC 4253) negotiates the algorithms and establishes an encrypted, integrity-checked channel using a Diffie-Hellman key exchange; the user authentication layer (RFC 4252) verifies the client’s identity, typically with a public key rather than a password; and the connection layer (RFC 4254) multiplexes the encrypted transport into multiple logical channels for shells, file transfers, and forwarded ports. Because SSH access is usually authenticated with long-lived key pairs rather than certificates that expire, an unmanaged SSH key estate is a common audit finding; this is precisely what SSH key lifecycle tooling is built to fix.

PGP and S/MIME: Encrypting and Signing Email

OpenPGP and S/MIME both encrypt and digitally sign email, but they build trust differently. OpenPGP, currently specified in RFC 9580 (which obsoletes the older RFC 4880), relies on a decentralized web of trust where users sign each other’s keys directly. S/MIME, specified in RFC 8551, instead relies on X.509 certificates issued by a CA, which makes it the more natural fit for organizations that already run a PKI for other purposes. Both protocols provide confidentiality and non-repudiation for a message body; neither one protects email headers, routing metadata, or messages once they are decrypted and sitting in an inbox.

WireGuard: A Lightweight Modern VPN Protocol

WireGuard is a newer VPN protocol built around a fixed, minimal cryptographic suite (Curve25519 for key exchange, ChaCha20-Poly1305 for encryption, BLAKE2s for hashing) instead of IPsec’s negotiable cipher list. It builds its handshake on the Noise Protocol Framework, and its full design is documented in its own technical whitepaper rather than an IETF RFC. Because it does not negotiate algorithms, there is no legacy-cipher fallback to misconfigure, which is a large part of why WireGuard configurations tend to be smaller and easier to audit than an equivalent IPsec setup. The tradeoff is less flexibility: if a component of WireGuard’s fixed suite is ever broken, the protocol needs a new version rather than a configuration change.

Kerberos: Network Authentication

Kerberos is a ticket-based authentication protocol, most familiar as the backbone of Windows Active Directory logons. A central Key Distribution Center authenticates a user once and issues a time-limited ticket, which the user then presents to individual services instead of re-authenticating to each one. Kerberos is built for trusted internal networks and Active Directory; it is not a substitute for TLS, IPsec, or SSH on traffic that crosses an untrusted network.

Which Encryption Protocol Should You Use for Each Use Case?

The right protocol follows directly from what you are protecting and who the endpoints are, not from which one is “strongest” in the abstract.

Use Case	Protocol	Current Recommended Version	Status
Web and API traffic (client-server)	TLS/SSL	TLS 1.3 (RFC 8446); TLS 1.2 acceptable minimum	TLS 1.0/1.1 and SSLv3 deprecated
Site-to-site VPN, network-layer tunnels	IPsec	IKEv2 (RFC 7296)	IKEv1 deprecated (RFC 9395)
Interactive remote admin, file transfer	SSH	SSH-2 (RFC 4251-4254)	SSH-1 obsolete, do not use
Email confidentiality, org-managed PKI	S/MIME	S/MIME 4.0 (RFC 8551)	Current
Email confidentiality, decentralized trust	OpenPGP	RFC 9580	Current; RFC 4880 obsoleted
Lightweight remote-access/site VPN	WireGuard	Fixed cipher suite per WireGuard whitepaper	Current
Internal AD authentication	Kerberos	Kerberos 5 (RFC 4120)	Current for trusted internal networks only

A few practical rules follow from this table. Public-facing web and API endpoints should terminate TLS 1.3 wherever client support allows, and fall back no lower than TLS 1.2. Site-to-site connections between data centers or cloud VPCs are usually a better fit for IPsec/IKEv2, since it works below the application layer and integrates with existing router and firewall infrastructure. Individual users connecting to internal resources from unmanaged networks are often better served by WireGuard or an IPsec client, not raw SSH tunneling, which is meant for administrative access rather than general-purpose networking. Choosing between S/MIME and OpenPGP for email usually comes down to whether the organization already operates a PKI: if it does, S/MIME reuses that investment; if the audience is external and decentralized, OpenPGP’s web of trust can be the more practical fit.

Threat Model and Deprecated Protocols to Avoid

Every protocol above assumes an active network adversary who can read, modify, inject, and replay packets, not just a passive eavesdropper. That assumption is why version negotiation itself is a security-critical piece of these protocols: an attacker who can force a downgrade to a weaker version can often defeat encryption without breaking any algorithm at all. Several protocol versions have been formally deprecated for exactly this reason and should be disabled everywhere they are still configured:

• SSLv3 was deprecated by RFC 7568 after the POODLE padding-oracle attack showed it could be reliably downgraded to and broken.
• TLS 1.0 and TLS 1.1 were deprecated by RFC 8996 in 2021, due to reliance on weak MD5/SHA-1 constructions and CBC-mode weaknesses with no viable secure configuration.
• IKEv1 was deprecated by RFC 9395 in 2023 alongside a set of obsoleted IPsec algorithms.
• SSH protocol version 1 has known cryptographic weaknesses and has been superseded by SSH-2 for close to two decades; any server still offering it is a critical finding in any assessment.

NIST’s guidance for federal TLS deployments, SP 800-52 Revision 2, requires TLS 1.2 as the floor and directs agencies toward TLS 1.3, reflecting the same logic: a protocol version stays acceptable only as long as no practical downgrade or cryptographic attack against it exists.

How to Harden Your Encryption Protocol Configuration

1. Inventory every protocol and version in use. Scan external and internal endpoints for the TLS versions, IKE versions, and SSH versions they currently accept, not just the version you intend to be running.
2. Disable deprecated versions at the configuration level. Remove SSLv3, TLS 1.0/1.1, IKEv1, and SSH-1 from server and load-balancer configs; do not rely on clients simply “not choosing” a weak option during negotiation.
3. Set a minimum accepted version, not just a preferred one. Configure TLS 1.2 as the floor with TLS 1.3 preferred, and confirm the server actually rejects earlier handshakes rather than just listing them as low-priority.
4. Restrict cipher suite negotiation to AEAD suites. Remove CBC-mode and RC4 suites from any TLS 1.2 configuration that must still support it; see the cipher suites guide for a scenario-based recommended list.
5. Verify certificate and key hygiene for each protocol. Confirm TLS and S/MIME certificates chain to a trusted root and are not near expiration, and that SSH authorized-key files do not contain orphaned or unmanaged keys.
6. Re-test after every change. Re-scan the same endpoints to confirm the deprecated version is actually gone from the server’s negotiated handshake, not just removed from a config file that was never reloaded.

Performance and Interoperability Trade-Offs

Protocol choice is rarely just about security margin; it also has real operational cost. TLS 1.3’s single-round-trip handshake reduces connection latency compared to TLS 1.2, which matters at the scale of millions of daily connections, but older clients, some legacy load balancers, and certain deep-packet-inspection middleboxes still only support TLS 1.2, forcing many public-facing services to keep a TLS 1.2 fallback path available. IPsec’s negotiable cipher list gives it broad interoperability across vendor hardware, at the cost of a larger attack surface from legacy algorithm options that must be actively pruned. WireGuard’s fixed cipher suite removes that negotiation overhead and keeps packet overhead low, which is why it performs well on mobile and constrained networks, but it cannot fall back to older algorithms if a client cannot support the current suite, and it lacks the vendor and appliance ecosystem IPsec has built up over two decades. SSH’s per-channel multiplexing is efficient for interactive sessions but was not designed for bulk network-layer encryption, which is why it should not be used as a general VPN replacement even though a “poor man’s VPN” over SSH tunneling is technically possible.

Key-Management Dependencies of Each Protocol

No protocol on this list is secure without a working key-management program behind it, and each one depends on a different kind of key infrastructure:

• TLS/SSL depends on a PKI that issues, renews, and revokes X.509 certificates before they expire or are compromised, work our CertSecure Manager platform automates across large certificate estates.
• IPsec depends on pre-shared keys or certificate-based authentication for its IKE phase, and on periodic rekeying to bound the amount of traffic protected by any single key.
• SSH depends on managing potentially thousands of long-lived, non-expiring key pairs across servers and service accounts, an estate that grows unmanaged far more easily than certificate-based access.
• PGP depends on individual users correctly managing their own private keys and a web of trust with no central revocation authority.
• S/MIME depends on the same PKI and CA infrastructure as TLS, extended to individual mailboxes.
• WireGuard depends on distributing and rotating Curve25519 key pairs per peer, typically outside of any centralized PKI.

In practice, the protocol is rarely where an encryption program fails; the key-management layer underneath it is. An organization can run TLS 1.3 everywhere and still be exposed if certificate renewal is manual and something expires unnoticed, or if SSH keys from an employee who left two years ago are still authorized on production servers.

Deployment Examples

A public e-commerce site terminates TLS 1.3 at its load balancer, backed by certificates issued and renewed automatically through a managed PKI, with TLS 1.2 kept available only for the small share of legacy clients that cannot negotiate 1.3. Two data centers connected by a permanent link run IPsec in tunnel mode with IKEv2, so every packet between them is encrypted at the network layer regardless of which application generated it. A platform team gives engineers SSH access to production hosts through a bastion host that issues short-lived certificates instead of static keys, closing the most common SSH key-sprawl finding. A legal team encrypts and signs client correspondence with S/MIME certificates issued from the firm’s own PKI, so recipients can verify the sender without exchanging keys manually. A remote workforce connects to the corporate network over WireGuard rather than a heavier IPsec client, trading configurable cipher agility for a smaller, faster, easier-to-audit connection.

Limitations

No encryption protocol protects data once it is decrypted at an endpoint; TLS protects data in transit, not the plaintext sitting in server memory or an application log afterward. Protocol strength is also relative to configuration: TLS 1.3 badly misconfigured (for example, with a weak or invalid certificate chain) offers less real protection than a correctly configured TLS 1.2 deployment. None of these protocols address key compromise after the fact, if a private key is stolen, every session it authenticated is retroactively suspect until that key is revoked and replaced. Finally, protocol support is not universal: legacy hardware, some IoT devices, and certain regulated environments may be unable to move to the current recommended version immediately, which makes a documented compensating-control and migration plan necessary rather than optional.

What Would Encryption Consulting Recommend?

Most organizations we work with do not have a protocol problem so much as a visibility and lifecycle problem: TLS 1.3 is the right choice almost everywhere, but the certificates behind it expire unnoticed, SSH keys accumulate with no owner, and IPsec configurations drift from their original hardening baseline over time. Our Encryption Advisory team starts every protocol review with a discovery pass across your environment, then maps findings to a remediation plan rather than a generic best-practices checklist. For the certificate estates behind TLS and S/MIME, CertSecure Manager automates issuance, renewal, and revocation so expiring certificates stop being a manual tracking exercise. For the PKI itself, our PKI Services team can design or rebuild the CA hierarchy and CP/CPS documentation that every certificate-based protocol on this page depends on. If your organization is still carrying SSLv3, TLS 1.0/1.1, or IKEv1 anywhere in production, that is not a future project, it is a current, exploitable gap worth closing this quarter.

FAQ

Is TLS the same thing as SSL?
No. SSL was TLS’s predecessor and every version of SSL, including SSLv3, is now deprecated (RFC 7568). “SSL” is still used informally to mean “TLS” in marketing and documentation, but no current deployment should actually be running the SSL protocol.

Should I disable TLS 1.2 now that TLS 1.3 exists?
Not necessarily. TLS 1.2, correctly configured with AEAD cipher suites, is still an accepted minimum under NIST SP 800-52 Rev. 2. Disable TLS 1.0/1.1 and SSLv3, but keep TLS 1.2 available as a fallback until you have confirmed every client you must support can negotiate TLS 1.3.

Is WireGuard secure enough to replace IPsec?
For many remote-access and site-to-site use cases, yes. WireGuard’s fixed, modern cipher suite and small codebase make it easier to audit than a full IPsec stack. The tradeoff is that IPsec has broader vendor and appliance support and configurable cipher agility that some regulated or legacy environments still require.

Why does SSH need three separate RFCs instead of one?
SSH’s transport, authentication, and connection functions are deliberately layered so each can evolve independently, RFC 4253 (transport), RFC 4252 (authentication), and RFC 4254 (connection) all sit on top of the base architecture in RFC 4251, letting authentication methods change without altering the encrypted transport underneath.

Does choosing the “most secure” protocol matter more than key management?
No. In our assessments, protocol misconfiguration and unmanaged keys or certificates cause far more real-world exposure than the choice of algorithm inside a modern protocol. A correctly managed TLS 1.2 deployment is safer in practice than a TLS 1.3 deployment with expired certificates or weak key storage.

Conclusion

Encryption protocols are the rulebooks that turn encryption algorithms into working security systems: TLS/SSL for client-server web traffic, IPsec for network-layer tunnels, SSH for remote administration, PGP and S/MIME for email, and WireGuard for lightweight modern VPNs. The right choice depends on the use case, not on which protocol sounds strongest, and every one of them depends on a key-management or PKI layer to actually deliver on its security promise. Keep deprecated versions, SSLv3, TLS 1.0/1.1, and IKEv1, out of production, set a documented minimum accepted version for each protocol you run, and treat certificate and key lifecycle management as part of the protocol, not an afterthought to it.

References

• IETF, RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3
• IETF, RFC 8996: Deprecating TLS 1.0 and TLS 1.1
• IETF, RFC 7568: Deprecating Secure Sockets Layer Version 3.0
• IETF, RFC 7296: Internet Key Exchange Protocol Version 2 (IKEv2)
• IETF, RFC 9395: Deprecation of the Internet Key Exchange Version 1 (IKEv1) Protocol and Obsoleted Algorithms
• IETF, RFC 4251: The Secure Shell (SSH) Protocol Architecture
• IETF, RFC 4253: The Secure Shell (SSH) Transport Layer Protocol
• IETF, RFC 9580: OpenPGP
• IETF, RFC 8551: S/MIME Version 4.0 Message Specification
• NIST, SP 800-52 Revision 2: Guidelines for TLS Implementations
• WireGuard, WireGuard: Next Generation Kernel Network Tunnel