Envelope Encryption: A Security Pattern for Multi-Tenant Systems
I keep finding teams that bolted encryption onto a multi-tenant app the wrong way — one global AES key in a config file, no per-tenant boundary, no real story for rotation, and no idea what they’d do if a key leaked. Envelope encryption is the pattern that makes the rotation story tractable and the per-tenant boundary mechanical, and once you’ve used it the design seems obvious. This is the version of the explanation I keep wanting to send people.
The shape of the pattern is simple. Encrypt your data with a Data Encryption Key (DEK) you generate per-record. Encrypt that DEK with a Key Encryption Key (KEK) that lives somewhere harder to reach. Store the encrypted data and the encrypted DEK side-by-side; the KEK never goes near the database.
Plaintext Data → [Encrypt with DEK] → Encrypted Data
DEK → [Encrypt with KEK] → Encrypted DEK
Storage: Encrypted Data + Encrypted DEK
To read:
Encrypted DEK → [Decrypt with KEK] → DEK
Encrypted Data → [Decrypt with DEK] → Plaintext Data
The whole thing turns on a separation that doesn’t sound like much: your data is encrypted with keys that are themselves encrypted. Every interesting property — rotation that doesn’t touch the data, blast-radius containment per tenant, a centralizable audit point — falls out of that one structural choice.
Why Bother#
The reason envelope encryption exists isn’t the one I see most often in introductory write-ups. It is not “AES is fast and RSA is slow, so encrypt with AES.” Every modern KMS — AWS KMS, GCP Cloud KMS, Azure Key Vault — uses a symmetric AES-256 KEK by default. Asymmetric keys are an option you have to opt into, not the thing you’re avoiding.
The actual reasons the pattern exists are a little less glamorous and a lot more operational.
Payload limits and per-call cost. AWS KMS caps Encrypt plaintext at 4 KB. Even if you ignored cost and latency, you couldn’t bulk-encrypt a row of any real size with the KMS API. Generating a DEK locally and encrypting just the DEK with the KMS gets around the limit, and gets around the per-call cost — KMS charges per request, and a per-row KMS call gets expensive fast.
Rotation that doesn’t re-encrypt the data. This is the real win. When you rotate a KEK, you decrypt the encrypted DEKs with the old KEK and re-encrypt them with the new one. The encrypted data is untouched. If you have terabytes of encrypted records and a few thousand DEKs per tenant, the rotation is a job that finishes in minutes instead of one that takes a maintenance window.
A per-tenant blast radius. In a multi-tenant system, each tenant gets their own KEK. A KEK compromise scopes to one tenant. Some compliance regimes (HIPAA BAAs, certain financial frameworks) want this anyway. Enterprise customers will eventually ask to bring their own KEK — AWS calls this “external key store” / XKS, GCP calls it EKM — and an envelope-encrypted system is one short step from supporting it. A non-envelope system isn’t.
The shape of that isolation is the part most diagrams of this pattern undersell:
graph TB
subgraph "Tenant A Data"
A1[Record 1
Encrypted Data A1
Encrypted DEK A1]
A2[Record 2
Encrypted Data A2
Encrypted DEK A2]
end
subgraph "Tenant B Data"
B1[Record 1
Encrypted Data B1
Encrypted DEK B1]
B2[Record 2
Encrypted Data B2
Encrypted DEK B2]
end
subgraph "Key Management Service"
KEK_A[KEK for Tenant A]
KEK_B[KEK for Tenant B]
end
A1 -.unwrap DEK.-> KEK_A
A2 -.unwrap DEK.-> KEK_A
B1 -.unwrap DEK.-> KEK_B
B2 -.unwrap DEK.-> KEK_B
style KEK_A fill:#e1f5ff
style KEK_B fill:#ffe1e1
style A1 fill:#e1f5ff
style A2 fill:#e1f5ff
style B1 fill:#ffe1e1
style B2 fill:#ffe1e1
The records are physically intermingled in the same database — one table, one schema, no per-tenant partitioning required. The cryptographic boundary between them lives entirely in the KEK column. Revoke a tenant’s KEK in your KMS and every record that belongs to them becomes mathematically unreadable, with no DELETE statements run, no backups updated, and no application code involved. That property is hard to get any other way.
An audit choke point. Every decryption requires a KMS call. The KMS already logs every call. You get tenant-scoped, per-operation auditability for free, without having to instrument your application code.
Where to Put the Encryption#
The first real architecture decision is where the encryption code runs. The choices in practice are in-process, behind a service, or a hybrid where a shared library encapsulates the work but each service still talks to KMS directly.
In-process keeps things flat. The application generates DEKs, calls KMS to wrap them, and stores the result. There’s no extra service to operate, no extra hop in the request path, and the failure modes are easy to reason about. The cost is that KEK access is now distributed across every service that touches sensitive data; the audit story is “go look at every service’s IAM role.” For a single small product, this is fine and probably right.
Behind a service — every component that needs to encrypt or decrypt calls a dedicated encryption service that owns the KMS client and the audit log — gives you exactly one place to enforce policy and exactly one IAM principal that can reach the KEK. Operationally, you’ve added a hop and an HA dependency, and the service has to scale with your encrypt/decrypt traffic, which is the same as your write/read traffic. This is the regime where regulated industries land, because the audit boundary is unambiguous and the security review is small.
The hybrid is what most growing teams I’ve seen actually run: a shared encryption library that every service imports, but with the KEK access centralized — usually by routing all kms:Decrypt through a single IAM role that’s assumed across the fleet. You get most of the centralization benefits without paying the latency tax of a separate service. The trade is that the library is now a security-critical dependency you have to keep current across services.
I’d start in-process and move to the hybrid the moment a second service needs to decrypt anything. Going straight to a service-from-day-one is usually overkill, but waiting until you have eight services and three KMS principals to centralize is painful.
Generating and Storing DEKs#
The canonical way to generate a DEK with AWS is to ask KMS to do it for you, with GenerateDataKey. This is the operation envelope encryption is named after — it returns both the plaintext DEK (so you can encrypt the data immediately) and the encrypted DEK (so you can throw the plaintext away the moment you’re done):
import boto3
kms = boto3.client('kms')
def new_dek(kek_id: str, tenant_id: str):
"""Ask KMS to mint a 256-bit DEK and wrap it for us in one call."""
resp = kms.generate_data_key(
KeyId=kek_id,
KeySpec='AES_256',
EncryptionContext={'tenant_id': tenant_id},
)
return resp['Plaintext'], resp['CiphertextBlob']
You can also generate the DEK locally with os.urandom(32) and call kms.encrypt() to wrap it — that flow is fine, but it’s two operations where the canonical one is one, and the only reason to reach for it is if you’re rotating a DEK you already have. For a fresh record, use GenerateDataKey.
GCP Cloud KMS doesn’t have a one-shot equivalent. You generate the DEK locally and call kms.encrypt() to wrap it. Same pattern, two calls instead of one.
What you store alongside the data:
{
"id": "record_123",
"tenant_id": "tenant_abc",
"encrypted_data": "base64...",
"encrypted_dek": "base64...",
"kek_id": "alias/tenant-abc/v5",
"algorithm": "AES-256-GCM",
"iv": "base64...",
"created_at": "2025-10-13T12:00:00Z"
}
Two pieces of metadata are load-bearing. kek_id records which KEK version wrapped this DEK; rotation needs it to know whether a record is on the old key or the new one. algorithm lets you change the data-encryption algorithm later without an everything-must-be-migrated event — your decrypt path branches on the field. The iv belongs there too, for reasons we’ll get to in a moment.
Where the KEK Lives#
Four reasonable options, in roughly the order I see teams pick them.
Cloud KMS is the default, and it should be. AWS KMS, GCP Cloud KMS, and Azure Key Vault all give you hardware-backed keys, IAM integration, automatic rotation, and audit logging without you writing anything. The downsides are real but bounded: vendor lock-in, per-operation cost, and rate limits that you will eventually hit if you’re decrypting on the read path of every request. (The fix for the rate limit is usually data-key caching — wrap a DEK once, reuse it for many records — not running away from KMS.)
Self-hosted HSMs are where you go when compliance demands it or when you have specific cryptographic mandates that the cloud KMS providers don’t meet. The cost is real money and real expertise; you don’t pick this for fun.
HashiCorp Vault is the option I want to be more explicit about, because most write-ups (mine included on a previous pass) bury it as a footnote. For non-cloud or hybrid deployments, Vault is the primary answer, not a fallback. The transit secrets engine gives you a KMS-shaped API — generate-data-key, encrypt, decrypt, rotation, audit — without depending on AWS, GCP, or Azure. You can run it in your own datacenter, on a customer’s VPC, in an air-gapped environment, or alongside cloud workloads when the cloud KMS isn’t acceptable for the data. The audit logging is excellent. The price you pay is operating Vault, including securing it — Vault is now a key-management system in its own right and needs its own threat model, its own backup story, and its own break-glass procedure.
The cases where I’d actively reach for Vault over cloud KMS:
- On-prem or hybrid deployments. Self-explanatory: you don’t have a cloud KMS to call. Vault is the encryption substrate that ships with your software.
- Sovereignty and data-residency requirements. Some regulators (and some enterprise customers) won’t accept that key material lives in a US cloud provider, full stop. Vault inside the customer’s network — or inside a region your customer controls — gets you past that conversation.
- Air-gapped or restricted-network environments. Defense, certain regulated finance, classified workloads. A KMS that requires outbound internet is a non-starter.
- You’re the vendor and your customers each run their own deployment. Vault travels with the software; cloud KMS doesn’t, or rather it tethers each install to a specific cloud provider in a way that won’t survive your enterprise procurement process.
- Multi-cloud with a single KEK boundary. Running on AWS, GCP, and Azure simultaneously and wanting one place to manage keys is genuinely awkward with cloud-native KMS. Vault gives you one.
For everything else — pure cloud deployments, single-tenant SaaS, anything where you’d rather not run another stateful service — start with cloud KMS. The operational savings dominate for years, and the migration to Vault later (if you ever need it) is mostly schema and code-wrapper work; the data on disk doesn’t have to move.
Application-level KEK storage rounds out the bottom of the list. The KEK lives in your secrets manager and the application reads it on startup. Lowest-friction option, the one I’d reach for in a side project, but the audit story is weak (your secrets manager logs reads, not uses) and rotation becomes a config-and-deploy event. Not appropriate for sensitive customer data; perfectly fine for, say, encrypting feature-flag overrides.
What the Two Paths Actually Look Like#
It’s worth being explicit about the round-trip. The encrypt path:
sequenceDiagram
participant App as Application
participant EncSvc as Encryption Service
participant KMS as KMS / KEK Store
participant DB as Database
App->>EncSvc: Encrypt(plaintext, tenant_id)
EncSvc->>KMS: GenerateDataKey(tenant_kek, AES_256, ctx={tenant_id})
KMS->>EncSvc: plaintext DEK + encrypted DEK
EncSvc->>EncSvc: AES-GCM(plaintext, plaintext DEK, fresh IV)
EncSvc->>EncSvc: drop plaintext DEK
EncSvc->>App: ciphertext + tag + IV + encrypted DEK + kek_id
App->>DB: store record
And the decrypt path, which is the one most people get bitten on — every read in your hot path turns into a KMS call unless you cache:
sequenceDiagram
participant App as Application
participant EncSvc as Encryption Service
participant KMS as KMS / KEK Store
participant DB as Database
App->>DB: fetch record
DB->>App: ciphertext + tag + IV + encrypted DEK + kek_id
App->>EncSvc: Decrypt(record, tenant_id)
EncSvc->>KMS: Decrypt(encrypted DEK, ctx={tenant_id})
Note over EncSvc,KMS: ctx mismatch fails the GCM tag,
not just a policy check
KMS->>EncSvc: plaintext DEK
EncSvc->>EncSvc: AES-GCM-Decrypt(ciphertext, DEK, IV, tag)
EncSvc->>EncSvc: drop plaintext DEK
EncSvc->>App: plaintext
The encryption context line in the decrypt sequence is doing a lot of work — that’s the section coming up next.
Encryption Context, Nonces, and the GCM Footguns#
This is the section the introductory posts skip and where most of the bugs hide.
Encryption context. AWS KMS lets you pass an EncryptionContext dictionary on encrypt and requires the same dictionary on decrypt. The framing in the AWS docs is “additional authenticated data,” which I think undersells it. The encryption context is included in the GCM authentication tag — a mismatched context doesn’t just fail a policy check, it fails the cryptographic decryption. If you encrypt a DEK with {tenant_id: "abc"} and someone tries to decrypt it with {tenant_id: "xyz"}, the decrypt call returns an InvalidCiphertextException. This means encryption context is a hard mechanical binding between a wrapped DEK and the metadata it should only ever be used with. Use it for tenant_id always, and for whatever else identifies the record uniquely (table name, record id) when you can.
Nonces are not optional. AES-GCM, which is what you should be using for the data-encryption step, requires a unique nonce per (key, plaintext) pair. Reusing a nonce with the same DEK is catastrophic — it doesn’t just leak data, it leaks the key stream, and an attacker who sees two ciphertexts encrypted under the same (key, nonce) pair can recover the XOR of the plaintexts directly. Generate a fresh 96-bit nonce for every encryption (os.urandom(12) in Python; crypto/rand.Read in Go), store it alongside the ciphertext, and never let a counter or a hash-of-something stand in. If your DEK is per-record and you generate a fresh nonce per record, you have ~2^48 records of headroom under the NIST SP 800-38D safety margin before you should rotate. That’s a lot — but it’s not infinity, and it’s worth being aware of if you’re caching DEKs.
Authentication tags. AES-GCM produces a 16-byte tag that authenticates both the ciphertext and the encryption context. Store it. Verify it. The most common mistake here is a library that returns ciphertext and tag separately and an integrator who only stores the ciphertext.
This stuff is buried in two paragraphs because it has to be — you can’t envelope-encrypt safely without internalizing it, and there’s no shortcut. If you’re rolling your own data-encryption step, use a high-level library that won’t let you skip the nonce or drop the tag (cryptography.hazmat.primitives.ciphers.aead.AESGCM in Python, crypto/cipher.NewGCM in Go). If you’re using a higher-level wrapper like AWS Encryption SDK, it handles all of this for you and you should probably just use it.
Keeping Keys Out of Logs#
A surprisingly large fraction of “encryption breach” post-mortems trace back to plaintext DEKs ending up in a structured log somewhere. The mitigation is layered, not single-shot.
The first layer is types. Wrap any plaintext key material in a type whose __repr__ and __str__ deliberately don’t expose it:
class SensitiveBytes:
"""Wrapper that prevents accidental logging of sensitive data."""
def __init__(self, data: bytes):
self._data = data
def __repr__(self):
return f"<SensitiveBytes length={len(self._data)}>"
def __str__(self):
return "<SensitiveBytes [REDACTED]>"
def reveal(self) -> bytes:
return self._data
This catches the dominant case, which is someone writing logger.info(f"got dek: {dek}") six months from now without thinking. It does not catch a determined developer who calls .reveal() and logs the result, and it does not catch a logging framework that introspects with __dict__. It is a cheap forcing function, not a hard guarantee.
The second layer is error-message hygiene. Every except block that touches encryption code should re-raise without the original error chained, and the message it does emit should not include any of the inputs:
def decrypt_record(record, tenant_id):
try:
dek = kms_decrypt(record.encrypted_dek, tenant_id)
return aes_gcm_decrypt(record.encrypted_data, dek, record.iv)
except DecryptionError as e:
logger.error(
"decryption_failed",
extra={
"tenant_id": tenant_id,
"record_id": record.id,
"error_type": type(e).__name__,
},
)
raise DecryptionError("Failed to decrypt record") from None
The from None matters: it suppresses the original exception’s chained context, which on some libraries includes parameters the underlying error captured. Treat raw exception objects as potentially-leaking and don’t pass them outward.
The third layer is structured audit logs. Log every encrypt/decrypt operation with tenant_id, kek_id, operation, success bit, and timestamp. Never log the encrypted DEK or any byproduct of the operation. Audit logs are read by people who don’t necessarily need access to keys; they should be able to see “this happened” without seeing the material.
I deliberately don’t recommend regex-based log redaction as a primary control. It is fragile (one new field name and your regex misses), brittle (multi-line JSON breaks naive patterns), and gives a false sense of security. As defense-in-depth at the log-aggregator boundary, it is fine. As your only line of defense, it is not.
The boundary-redaction story has gotten meaningfully better recently, though, and worth flagging: OpenAI’s Privacy Filter, released in April 2026, is a 1.5B-parameter open-weight model (Apache 2.0, ~50M active parameters) trained specifically for context-aware PII detection in unstructured text. It runs locally — laptop or browser — so the logs themselves don’t have to leave your infrastructure to be sanitized, and the eight categories it detects include secrets (API keys, passwords) alongside the usual names/addresses/emails/phone-numbers. For anyone shipping a log-redaction pipeline, this is the first off-the-shelf option I’ve seen that’s actually plausible at the aggregator boundary; it’s the upgrade path away from regex without the “pipe your logs through a third-party API” trade-off. Same caveats as before: defense in depth, not your primary control. The structural mitigations above are still the load-bearing ones.
I also deliberately don’t recommend trying to zero plaintext keys from Python memory. The standard ctypes.memset(id(data), 0, …) recipe you’ll find on Stack Overflow is buggy — id(data) is the address of the PyObject header, not the buffer — and even if you fix it, the garbage collector and the string-interning machinery have likely produced copies you can’t reach. If you genuinely need to zero memory, do the encryption step in a language that lets you (Rust, Go, C with explicit_bzero) and call into it from Python.
Rotation#
KEK rotation is the easy case. Decrypt every encrypted DEK with the old KEK, re-encrypt with the new one, write back. The encrypted data is untouched.
def rotate_kek(tenant_id, old_kek_id, new_kek_id):
cursor = db.scan(
"SELECT id, encrypted_dek FROM records "
"WHERE tenant_id = ? AND kek_id = ?",
tenant_id, old_kek_id,
)
for record in cursor:
dek = kms.decrypt(
CiphertextBlob=record.encrypted_dek,
EncryptionContext={'tenant_id': tenant_id},
)['Plaintext']
new_blob = kms.encrypt(
KeyId=new_kek_id,
Plaintext=dek,
EncryptionContext={'tenant_id': tenant_id},
)['CiphertextBlob']
db.update(
"UPDATE records SET encrypted_dek = ?, kek_id = ? WHERE id = ?",
new_blob, new_kek_id, record.id,
)
Run it in batches with a checkpoint. Don’t delete the old KEK when you finish — keep it around in a “scheduled for deletion” state for at least the duration of your backup retention window, in case you have to roll back.
The shape of the rotation, with explicit per-record steps:
sequenceDiagram
participant Op as Operator
participant Rot as Rotation Job
participant KMS as KMS / KEK Store
participant DB as Database
Op->>KMS: create KEK v2 for tenant
Op->>Rot: rotate(tenant_id, kek_v1 → kek_v2)
loop for each record at kek_v1
Rot->>DB: fetch encrypted_dek
DB->>Rot: encrypted_dek
Rot->>KMS: Decrypt(encrypted_dek, kek_v1, ctx)
KMS->>Rot: plaintext DEK
Rot->>KMS: Encrypt(plaintext DEK, kek_v2, ctx)
KMS->>Rot: encrypted_dek'
Rot->>DB: update encrypted_dek, kek_id = v2
Note over Rot,DB: ciphertext + tag + IV unchanged
end
Rot->>Op: rotation complete
Op->>KMS: schedule kek_v1 deletion (after backup retention)
Note what’s not happening: the encrypted data, the IV, and the GCM tag never leave the database. The whole job is a re-wrap of small DEK ciphertexts. On a tenant with millions of records and thousands of distinct DEKs, this finishes in minutes; the equivalent without envelope encryption is “rewrite every row,” which is a maintenance window.
DEK rotation is the expensive case, because it requires re-encrypting the data itself. You usually don’t need to do this except in two situations: you’ve crossed a NIST nonce-safety budget on a particular DEK (rare for per-record DEKs, more common for cached DEKs), or you’ve had a key compromise and need to make the old ciphertexts unrecoverable. For the first, do it lazily — every read that touches an old-DEK record re-encrypts on the way out. For the second, you have an incident, not an engineering problem, and you should be doing it as fast as you can.
The application code shouldn’t care which version of the KEK encrypted a given record. Wrap the KMS calls in a small service object whose decrypt path reads kek_id from the record itself and routes to the right key. The encrypt path always uses the current KEK alias, which is the cleanest way to keep new writes from accidentally hitting an old key:
class EnvelopeCrypto:
def __init__(self, current_kek_alias):
self._current = current_kek_alias
def encrypt(self, plaintext, tenant_id):
plaintext_dek, encrypted_dek = new_dek(self._current, tenant_id)
iv = os.urandom(12)
ciphertext, tag = aes_gcm_encrypt(plaintext_dek, plaintext, iv)
return Record(
ciphertext=ciphertext, tag=tag, iv=iv,
encrypted_dek=encrypted_dek, kek_id=self._current,
)
def decrypt(self, record, tenant_id):
dek = kms.decrypt(
CiphertextBlob=record.encrypted_dek,
EncryptionContext={'tenant_id': tenant_id},
)['Plaintext']
return aes_gcm_decrypt(dek, record.ciphertext, record.iv, record.tag)
This is also the layer at which DEK caching belongs, if you decide you need it. Keep a TTL’d in-memory cache of (kek_id, encrypted_dek) → plaintext_dek, with a small bound on size, and you eliminate the per-request KMS call on hot reads. The cost is a memory-resident plaintext DEK; the bound is how many you’re willing to hold and for how long.
When You’d Reach For Something Else#
Envelope encryption isn’t the only shape this problem can take, and there are two adjacent patterns worth being explicit about — both of which I see people reach for instead, sometimes correctly and sometimes not.
Encryption-as-a-service — the Vault transit secrets engine, AWS Encryption SDK with raw kms.encrypt against arbitrary plaintext, GCP Tink — collapses the whole envelope into a service call. You hand the service plaintext, you get ciphertext back; you hand it ciphertext, you get plaintext. There is no DEK in your code, no IV in your schema, no AES-GCM library to integrate. The trade is a network call on every encrypt and every decrypt, plus the same per-call cost properties that pushed you toward envelope encryption in the first place. This is a great fit for low-volume, high-value data — per-tenant API credentials, signed token payloads, the kind of thing you encrypt once and decrypt occasionally. It is a poor fit for arbitrary user data on the read path, where the per-request KMS call dominates your latency budget. The cleanest signal for which one you want: if you’d consider DEK caching, you want envelope encryption; if you wouldn’t, encryption-as-a-service is probably the right call.
Per-value secret storage is the related but distinct pattern of treating each sensitive value as a named entry in a secrets manager — Vault KV, AWS Secrets Manager, GCP Secret Manager — and reading it on demand. This works for things that are sensitive and singular and rarely accessed: third-party API keys, per-tenant signing secrets, OAuth client secrets, the database password your service reads at startup. Secrets managers give you versioning, IAM, audit, and a clean rotation story for free. They are not designed for millions of lookups per second, are not priced for it, and are not structured for query-shaped access. The line I draw is roughly: if you’d ever join the value in a SQL query, it doesn’t belong in a secrets manager. If it’s a singleton or a small set of named things the application asks for by name, it does.
These aren’t either/or with envelope encryption. A real production system tends to use all three — envelope encryption for per-row sensitive fields, an encryption service for low-volume credential payloads that pass through the API surface, and per-value secrets for the genuinely-singleton things the application needs at startup. The mistake to avoid is forcing every shape of sensitive data through one of them.
Where I Land#
If you’re shipping a multi-tenant system that holds anything sensitive, the question isn’t should you envelope-encrypt. It’s what’s stopping you from envelope-encrypting? The pattern is straightforward, the cloud KMS providers have done most of the work, and the operational properties — rotation that doesn’t touch the data, per-tenant blast radius, a built-in audit log — are the kind of thing you only get to design in once.
The hard parts of envelope encryption aren’t the encryption. They’re the parts around it: nonce handling, encryption context, key versioning, audit hygiene, and not leaking plaintext DEKs through your logging stack. Get those right and the rest of the pattern carries you a long way.
Where I’d start: in-process encryption, AWS or GCP KMS for the KEK, GenerateDataKey for DEK creation, AES-GCM for the data step, encryption context bound to tenant_id, and kek_id/algorithm/iv stored in the schema from day one. That schema is the part that’s painful to add later, and the part that buys you almost everything else.
Where I’d avoid going: rolling the data-encryption step yourself with raw block ciphers, treating regex log-redaction as a primary control, or trying to zero plaintext keys from Python memory. None of those buy what they look like they buy.
Have experience with envelope encryption in production, or a footgun I missed? I’d love to hear it. Reach out on twitter @nrmitchi or shoot me an email.