AES encryption explained: how it works and how to use it
How AES works, what 25 years of cryptanalysis found, which modes to use, and the mistakes (nonce reuse, padding oracles, hardcoded keys) that break it.
By MD-5 Updated 7 min read
The Advanced Encryption Standard (AES) encrypts almost every HTTPS connection, disk and database you use. After 25 years of public cryptanalysis, the best known attack on the full cipher is only marginally faster than trying every key. When AES-protected data is exposed, the cause is almost always how AES was used: the wrong mode, a repeated nonce, a key in source code.
This guide covers how AES works, what the research says about its strength, which modes to choose, and the implementation mistakes that turn a strong cipher into a weak system. New to the topic? Start with the beginner’s guide to symmetric encryption.
Where AES came from
By the mid-1990s the Data Encryption Standard (DES) was visibly failing: its 56-bit key was within reach of purpose-built hardware. In January 1997 NIST announced an open, international competition for a replacement. Fifteen candidates were accepted in 1998, five reached the final round in 1999 (MARS, RC6, Rijndael, Serpent and Twofish), and in October 2000 NIST selected Rijndael, designed by Belgian cryptographers Joan Daemen and Vincent Rijmen.
It was published as FIPS 197 (opens in a new tab) in November 2001. The open process mattered: every candidate was attacked in public by the other teams, which is a large part of why AES is trusted today.
The parameters
- Block size: 128 bits, fixed.
- Key sizes: 128, 192 or 256 bits.
- Rounds: 10, 12 or 14 respectively.
- Structure: a substitution-permutation network, which transforms the whole block in every round (unlike DES’s Feistel network, which transforms half).
The 128-bit block matters as much as the key. Ciphers with 64-bit blocks, such as 3DES and Blowfish, start leaking information after a few tens of gigabytes under one key. AES’s block pushes that limit far beyond practical volumes.
How each round works
AES arranges the 16-byte block as a 4 × 4 grid of bytes (the state) and applies four operations per round:
- SubBytes: each byte is replaced using a fixed substitution table (the S-box). This is the only non-linear step and provides confusion.
- ShiftRows: each row of the grid is rotated left by 0, 1, 2 or 3 bytes.
- MixColumns: each column is multiplied by a fixed matrix, so every output byte depends on all four input bytes in its column. With ShiftRows, this provides diffusion: after two rounds, every output bit depends on every input bit.
- AddRoundKey: the state is XORed with a round key derived from the main key by the key schedule.
The final round omits MixColumns, which adds no security at the end and lets decryption mirror encryption. Decryption applies the inverse operations in reverse order.
What the cryptanalysis found
No practical attack on full AES has been published. The results that exist are worth knowing precisely, because they’re often exaggerated:
- Biclique attacks (Bogdanov, Khovratovich and Rechberger, 2011) recover a key in about 2^126.1 operations for AES-128 and 2^254.4 for AES-256. That’s a factor of roughly four faster than brute force: an academic result with no practical effect.
- Related-key attacks on AES-256 (Biryukov and Khovratovich, 2009) are faster than brute force, but only if the attacker can get data encrypted under several keys with a specific mathematical relationship. Properly generated random keys never have that relationship.
- Side-channel attacks are the real-world risk. In 2005 Daniel Bernstein showed that cache-timing differences in table-based software AES could leak the key over a network. Hardware AES instructions (Intel’s AES-NI, from 2010, and ARMv8 cryptography extensions) run in constant time and removed that class of leak on modern CPUs. Constant-time software implementations do the same where hardware support is missing.
The conclusion from the research is consistent: the algorithm holds, and the risk sits in the code around it.
Modes of operation
AES encrypts one 16-byte block. A mode of operation defines how to encrypt anything longer. NIST standardizes five classic modes in SP 800-38A (opens in a new tab) and GCM in SP 800-38D (opens in a new tab).
| Mode | How it works | Integrity? | Verdict |
|---|---|---|---|
| ECB | Each block encrypted independently | No | Never use: identical blocks give identical ciphertext |
| CBC | Each block XORed with the previous ciphertext | No | Legacy; needs a separate MAC |
| CFB / OFB | Turn the block cipher into a stream | No | Legacy |
| CTR | Encrypts a counter to make a keystream | No | Fast, but needs a separate MAC |
| GCM | CTR plus an authentication tag | Yes | Default choice for new code |
ECB is weak enough that NIST’s draft revision of SP 800-131A (Rev. 3) (opens in a new tab) proposes retiring it as a confidentiality mode.
GCM is authenticated encryption: it both hides the data and detects any change to it before decryption. It’s what TLS 1.3 uses by default. Its one hard rule is never reuse a nonce with the same key. SP 800-38D limits random 96-bit nonces to 2^32 messages per key for this reason.
The mistakes that break AES
Nonce reuse in GCM
Reusing a GCM nonce under the same key leaks the XOR of the two plaintexts and, worse, lets an attacker recover the authentication key and forge messages. Antoine Joux described this “forbidden attack” in 2006. In 2016 Böck and colleagues scanned the internet and found 184 HTTPS servers repeating GCM nonces (opens in a new tab), fully breaking the integrity of their connections, and over 70,000 more using random nonces in a way that risked it.
Fix: use your library’s nonce generation, never a counter you manage by hand across processes, and rotate keys well before 2^32 messages.
Padding oracles in CBC
Serge Vaudenay showed in 2002 that if a server decrypts attacker-supplied CBC ciphertext and reveals whether the padding was valid (through a distinct error, status code or timing), the attacker can decrypt the data byte by byte without the key, and often forge new ciphertext. The attack has been found repeatedly in production, including in ASP.NET in 2010 (opens in a new tab), where it exposed encrypted view state and authentication data.
Fix: use authenticated encryption. If you must use CBC, apply an HMAC over the IV and ciphertext and verify it before decrypting (encrypt-then-MAC).
Encrypted but not authenticated tokens
Encryption hides a token’s contents; it doesn’t stop tampering. Flipping bits in CTR or CBC ciphertext flips predictable bits in the plaintext, which can turn role=user into something else. Session tokens, password reset links and “encrypted” IDs built on unauthenticated modes are a recurring finding in application tests.
Hardcoded and derived keys
A key in a JavaScript bundle, mobile app binary or public repository means anyone can decrypt. Mobile apps are frequent offenders because decompiling makes the “hidden” key visible. A human-chosen string used directly as a key is almost as bad: MYSECRETKEY12345 is 128 bits long but has far less than 128 bits of unpredictability.
Fix: generate keys from a cryptographically secure random generator, store them in a key-management service or HSM, and if a key must come from a password, derive it with Argon2id or PBKDF2.
Static or predictable IVs
A CBC IV of all zeros, or one derived from a record ID, makes encryption deterministic: two records that start the same encrypt the same, which leaks information.
How to use AES correctly
- AES-GCM (or ChaCha20-Poly1305) through a vetted library: libsodium, Google Tink, or your platform’s standard crypto API. Never implement AES or a mode yourself.
- Random keys, stored in a KMS or HSM, never in code, config files or container images.
- Never reuse a nonce with the same key, and let the library generate nonces.
- Record the key version with each ciphertext, so keys can be rotated without guesswork.
- AES-256 for data that must stay confidential for decades. Grover’s quantum search algorithm would, at best, halve effective key strength, which leaves AES-256 with a comfortable margin. The NSA’s CNSA 2.0 suite (opens in a new tab) specifies AES-256 for this reason.
Where AES is used
- TLS/HTTPS: most connections use AES-GCM; the rest mostly use ChaCha20-Poly1305.
- Wi-Fi: WPA2 and WPA3 use AES-CCMP (and GCMP in WPA3’s higher modes).
- Disk encryption: BitLocker, FileVault and LUKS default to AES in XTS mode, a mode designed for storage.
- Messaging: the Signal Protocol, used by Signal and WhatsApp, encrypts messages with AES-256 in CBC mode with HMAC-SHA256, a correct encrypt-then-MAC construction.
- Cloud KMS: AWS KMS and Google Cloud KMS encrypt data with AES-256-GCM for symmetric keys.
For a direct comparison with its predecessor, see AES vs DES.
What a security test checks
If your application encrypts data itself, a web application penetration test looks for exactly the mistakes above: tokens that can be tampered with, decryption errors that act as padding oracles, keys recoverable from client-side code, and deterministic encryption that leaks patterns.
Sources
- NIST, FIPS 197: Advanced Encryption Standard (opens in a new tab), 2001 (updated 2023).
- NIST, SP 800-38A: Recommendation for Block Cipher Modes of Operation (opens in a new tab) and SP 800-38D: Galois/Counter Mode (opens in a new tab).
- NIST, SP 800-131A Rev. 3 (initial public draft) (opens in a new tab), 2024.
- A. Bogdanov, D. Khovratovich, C. Rechberger, Biclique cryptanalysis of the full AES (opens in a new tab), ASIACRYPT 2011.
- A. Biryukov, D. Khovratovich, Related-key cryptanalysis of the full AES-192 and AES-256 (opens in a new tab), ASIACRYPT 2009.
- D. J. Bernstein, Cache-timing attacks on AES (opens in a new tab), 2005.
- H. Böck, A. Zauner, S. Devlin, J. Somorovsky, P. Jovanovic, Nonce-disrespecting adversaries: practical forgery attacks on GCM in TLS (opens in a new tab), USENIX WOOT 2016.
- S. Vaudenay, Security flaws induced by CBC padding, EUROCRYPT 2002.
Published by
MD-5 · Certified cybersecurity services
Articles are researched and written in-house and link to their primary sources (standards, advisories, papers and public datasets) so you can check every claim. Spotted an error? Email contact@md-5.com and we’ll correct it.