Disk encryption was not the boundary we needed

The event platform already protected transport with TLS and relied on encrypted infrastructure storage. Those controls matter, but they answer different questions.

Volume encryption protects lost disks and snapshots. Database transport encryption protects network traffic. Neither prevents a database reader, overly broad reporting account, accidental export, or plaintext backup workflow from seeing application values after the database has legitimately opened its storage.

Dynamic event forms can contain names, contact details, arrival plans, accessibility information, or other contextual answers. The safer design was to encrypt those answers at the application boundary before SQL persistence.

Infrastructure encryption protects storage media. Application-level encryption protects selected fields from infrastructure paths that never need their plaintext.

The design used KMS-backed envelope encryption: a managed root key protects short-lived data keys, and those data keys protect the actual answers.

Concepts in plain language

  • KMS: a managed key service that controls a root encryption key without handing that root key to the application.
  • Data encryption key, or DEK: a short-lived key used by the application to encrypt one bounded group of answers.
  • Envelope encryption: encrypting data with a DEK, then encrypting that DEK with the stronger managed root key.
  • AES-256-GCM: a data-encryption algorithm that both hides the value and detects modification.
  • AAD or encryption context: non-secret identity information that acts like a cryptographic label, binding ciphertext to its intended purpose and owner.

Step 1: choose the encryption unit

One KMS request per field would be simple but expensive and slow. One data key for the entire database would create an enormous blast radius. The useful middle ground was the response owner: one reservation or direct participation.

For each owner group, the service requested one AES-256 data key and encrypted every answer in that group with the same data key but a unique initialization vector.

KMS root key
  -> encrypted data key for participation A
       -> answer 1 ciphertext + IV + tag
       -> answer 2 ciphertext + IV + tag
       -> answer 3 ciphertext + IV + tag

  -> encrypted data key for participation B
       -> answer 1 ciphertext + IV + tag

This limited a data key to one business aggregate while keeping KMS calls bounded. A read operation could unwrap the key once and decrypt the owner’s answers in memory.

Step 2: generate a data key instead of downloading the root key

The application never received the KMS root key. It called GenerateDataKey with AES_256 and received two related values:

  • a 32-byte plaintext data encryption key, or DEK;
  • the same DEK encrypted under the managed KMS key.

The plaintext DEK existed only long enough to perform local AES-GCM operations. The encrypted DEK was safe to store beside the ciphertext.

GenerateDataKey
  -> Plaintext DEK       use briefly, then erase best-effort
  -> CiphertextBlob DEK  store in the database

On a later read, the service sent the encrypted DEK to KMS Decrypt, reconstructed the same context, and received the plaintext DEK for the minimum necessary duration.

Why not call KMS Encrypt for every answer? KMS is designed to protect small key material and secrets, not to replace a bulk data cipher for every application value. Envelope encryption keeps large or frequent data operations local while centralizing root-key policy and audit.

Step 3: use AES-256-GCM correctly

Each answer used:

  • AES with a 256-bit data key;
  • Galois/Counter Mode;
  • a fresh cryptographically random 12-byte IV;
  • an authentication tag stored separately;
  • associated authenticated data derived from immutable owner context.

The plaintext value was first serialized into a canonical typed representation. This preserved the distinction between text, numbers, booleans, arrays, objects, and null where the domain allowed them.

ciphertext, tag = AES-256-GCM(
  key = plaintext DEK,
  iv = fresh random 96-bit value,
  plaintext = canonical JSON value,
  aad = protocol version + scope type + owner identifier
)

The IV must never repeat for the same key. Generating a fresh random IV per answer is therefore mandatory even when several answers share one owner DEK.

GCM also authenticates the ciphertext and associated data. A changed ciphertext, tag, IV, or owner context causes decryption to fail instead of returning corrupted plaintext.

Step 4: bind ciphertext to its owner with associated data

Encryption alone does not stop a valid ciphertext row from being copied to a different participation. If the key material is also copied, naive decryption may still succeed.

Associated authenticated data, or AAD, bound each answer to its intended scope:

event-form-response:v1:<scopeType>:<ownerId>

Moving the row to another owner changed the reconstructed AAD, so authentication failed.

The KMS call used an encryption context containing only non-secret descriptors such as:

service = event-platform
purpose = event-form-response
scopeType = participation
ownerId = internal numeric owner id
version = 1

AWS documents that KMS encryption context is cryptographically bound to the ciphertext and can be used in key policies and grants. It is also logged, so it must never contain an answer, email address, name, token, or other sensitive value.

Using context in both layers gave two useful properties:

  • the local answer ciphertext could not be transplanted between owners;
  • the encrypted DEK could not be unwrapped under a mismatched KMS context.

Step 5: design a versioned storage envelope

The database needed enough metadata to decrypt a value years later without guessing which scheme created it.

The response tables gained nullable columns like:

encryption_version
encryption_key_id
encrypted_data_key
value_ciphertext
value_iv
value_auth_tag

The key identifier recorded the KMS key reference returned or resolved at encryption time. This matters during key rotation: new writes can use a new root key while old rows remain decryptable with the key recorded in their envelope.

Versioning was not optional. Cryptographic formats evolve. A future version may change serialization, context construction, algorithms, or key providers. encryption_version = 1 makes that transition explicit.

Plaintext and encrypted state were mutually exclusive

During migration, a row could be one of two valid shapes:

Legacy row
  plaintext value present
  all envelope fields null

Encrypted v1 row
  plaintext columns null
  complete envelope metadata present

Database check constraints rejected half-encrypted rows. IV and tag lengths were checked. This prevented an application bug from writing ciphertext without the information needed to open it.

Step 6: encrypt every write path, not only creation

Dynamic answers appeared in more places than the primary create operation:

  • reservation form responses;
  • direct participation responses;
  • operator amendments;
  • previous and next values in revision history;
  • cleanup and anonymization updates;
  • read projections used by authorized operators.

Encrypting only new registration would have left amended values or revision history in plaintext. The repository boundary handled all of them.

For an amendment, the flow became:

  1. Load and decrypt the current answer under the existing envelope.
  2. Validate the requested change.
  3. Generate an envelope for the new current value.
  4. Encrypt both previous and next revision values.
  5. Store current answer and revision atomically.
  6. Keep audit metadata free of answer content.

Cleanup explicitly nulled plaintext, ciphertext, IV, tag, encrypted data key, key identifier, and version metadata. Leaving an encrypted answer after its retention deadline would still be retaining the answer.

Step 7: group decryptions without weakening isolation

A list view may return many answers for the same owner. Calling KMS once per row creates unnecessary latency and cost.

The decryptor grouped rows by:

  • scope type;
  • owner identifier;
  • encryption version;
  • KMS key identifier;
  • encrypted data key.

Each group required one KMS unwrap, followed by local AES-GCM operations. Groups never crossed owner boundaries, even if malformed data claimed to share an encrypted key.

The plaintext DEK buffer was overwritten in a finally block after use. Managed runtimes cannot guarantee that every copy disappears immediately, but a best-effort zeroization reduces accidental lifetime and makes the intended handling explicit.

Step 8: fail closed without leaking KMS details

KMS is a network dependency. The service used a bounded timeout and a small retry budget. It did not retry indefinitely inside an HTTP transaction.

When encryption was required and KMS was unavailable, the write failed. Falling back to plaintext would silently violate the privacy contract.

When an encrypted value could not be opened, the read failed with a stable service error. Raw provider messages, key identifiers, ciphertext, encryption context, and SDK stack traces did not reach the browser.

Operational telemetry recorded only safe information:

  • operation type;
  • mode;
  • success or failure class;
  • latency;
  • count of rows or owner groups;
  • stable request correlation identifier.

No answer, plaintext key, cookie, bearer token, or encrypted blob belonged in logs.

Step 9: test cryptographic properties, not implementation trivia

The focused tests used a controlled KMS mock and proved behavior:

  • disabled mode preserves the legacy representation;
  • write mode requests an AES-256 data key;
  • plaintext answer is absent after encryption;
  • decrypt returns the original typed value;
  • each answer receives a unique IV;
  • KMS context contains only the expected non-secret scope;
  • moving ciphertext to another owner fails authentication;
  • multiple answers sharing one envelope require one unwrap;
  • a stored historical key identifier remains usable after active-key rotation;
  • tampered tag, IV, ciphertext, or encrypted key fails closed;
  • provider errors are translated into stable application errors.

Repository tests then verified create, amend, list, revision, and cleanup paths. Migration tests verified envelope completeness constraints and changelog reachability.

What envelope encryption changes operationally

The database no longer contains directly readable answers, but the application still can decrypt them when authorized. This is deliberate. Encryption is not a substitute for authorization, least privilege, retention, or audit.

It adds a new dependency and new failure modes:

  • KMS permissions must be narrowly scoped;
  • key deletion can make data permanently unreadable;
  • context construction must stay stable;
  • old key identifiers must survive rotation;
  • backups must include complete envelope metadata;
  • rollback must understand which rows are already encrypted.

Those constraints make deployment strategy part of the cryptographic design. The final article explains the phased path: Rolling Out Encryption Safely with Liquibase, Backfill, and K3s.

This article intentionally omits organization names, domains, credentials, infrastructure addresses, and identifying production details. The architecture and lessons remain real.