Cryptography was not the riskiest part of the release

The AES-GCM and KMS operations were focused and testable. The harder problem was introducing them into a live system that already contained plaintext answers and had multiple application replicas, background cleanup, migrations, and rollback expectations.

A single “enable encryption” switch would have coupled too many irreversible changes:

  • schema creation;
  • application compatibility;
  • key and identity provisioning;
  • new encrypted writes;
  • legacy reads;
  • historical backfill;
  • key rotation;
  • rollback.
A safe encryption migration is a compatibility program, not a deployment toggle.

The rollout separated those concerns into observable phases. Every phase had an entry gate, verification, stop condition, and rollback boundary.

Concepts in plain language

  • Liquibase: a tool that applies versioned database schema changes in a repeatable order.
  • Backfill: a controlled job that converts rows already stored in the old format into the new format.
  • GitOps: keeping the desired deployment state in Git and letting an automated controller reconcile the cluster to that reviewed state.
  • K3s: a lightweight Kubernetes distribution; Kubernetes schedules and supervises the application containers.
  • ClusterIP: a private service address reachable inside the cluster or from an attached node, but not intended as a public website address.
  • Canary: a deliberately small first use of a change that proves behavior before normal traffic receives it.

Step 1: add a backward-compatible schema first

The Liquibase changeset added nullable envelope columns while preserving legacy plaintext columns. It did not rewrite existing rows during schema migration.

This mattered for three reasons:

  1. The migration stayed fast and predictable.
  2. Old application Pods could continue reading and writing during a rolling deployment.
  3. KMS availability was not required to complete DDL.

The migration also added check constraints that accepted exactly two shapes: complete legacy state or complete encrypted version-one state. Partial envelopes were invalid.

Indexes selected legacy rows that still required backfill. They were partial and operationally focused rather than added to every read path.

Phase-one migration
  -> add nullable envelope columns
  -> add integrity constraints
  -> add backfill candidate indexes
  -> do not encrypt or erase existing values

Liquibase rollback could drop the new schema only before encrypted writes existed. Once plaintext columns had been cleared, blindly dropping envelope columns would destroy the only readable representation. The runbook said this explicitly instead of offering a comforting but unsafe rollback command.

Step 2: deploy code with encryption disabled

The application supported three modes:

  • disabled: read and write the legacy representation;
  • read-only: read both legacy and encrypted rows, but continue legacy writes;
  • read-write: read both formats and encrypt every new or changed answer.

The first application rollout used disabled.

This proved that:

  • every Pod understood the new columns;
  • entity and repository mappings were correct;
  • cleanup could handle envelope metadata;
  • the migration was reachable in every environment;
  • no KMS dependency had entered the request path yet.

If this phase failed, ordinary deployment rollback was still safe because no encrypted-only rows existed.

Step 3: provision secrets before they are referenced

The Git repository contained only secret contracts: expected names and keys, never values. Example manifests were deliberately excluded from active Kustomize resources.

The runtime needed two independent secret groups:

  • BFF session keyring for the frontend server;
  • KMS access configuration for the backend workload, unless workload identity supplied it without static credentials.

Kubernetes documentation warns that Secret values are base64-encoded, not encrypted, and recommends least-privilege access plus encryption at rest. See Good practices for Kubernetes Secrets and Encrypting Confidential Data at Rest.

The operational rules were:

  • never commit a real key, credential, or rendered Secret;
  • prefer workload identity or an external secret provider over long-lived static credentials;
  • grant only GenerateDataKey and Decrypt on the intended key and context;
  • restrict Secret get, list, and watch permissions;
  • make every BFF replica share the same ordered keyring;
  • verify Secret presence without printing its content;
  • stop rollout if the required key or identity is absent.

Marking a Secret reference as optional can be useful during compatibility rollout, but production activation must still have a hard gate. “Optional in YAML” must not mean “silently insecure in the application.”

Step 4: move to read-only compatibility

After every Pod ran compatible code and KMS access was proven, the backend moved to read-only mode.

In this phase:

  • legacy rows remained readable without KMS;
  • encrypted test rows could be decrypted;
  • ordinary writes remained plaintext;
  • operators could validate permissions, latency, context, and error handling without changing the main write contract.

A controlled canary created a test envelope, opened it through the normal repository, and removed it through the normal cleanup path. The test used synthetic data and did not log ciphertext or context values.

Useful gates included:

  • all API replicas ready on the expected image digest;
  • schema migration complete;
  • KMS call success from the workload identity;
  • correct region and key identifier;
  • no unexpected provider retries or timeout growth;
  • legacy and encrypted read tests both green;
  • backup and restore procedure tested with envelope metadata.

Step 5: enable encrypted writes only after every reader is ready

The switch to read-write happened only when all processes that could read form responses understood encrypted rows. This included API replicas, workers, cleanup jobs, administrative views, export paths, and one-off operational scripts.

The first write canary verified:

  1. The request completed normally.
  2. Plaintext columns were null.
  3. Envelope columns were complete.
  4. The application could read the answer back.
  5. A different owner context could not decrypt it.
  6. Logs and traces contained no answer or key material.
  7. Cleanup erased the complete envelope.

Only then did normal traffic receive encrypted writes.

The rollout watched request errors, KMS latency, KMS throttling, database constraint failures, Pod restarts, and memory behavior. A crypto feature that is correct but causes request queues to grow is not production-ready.

Step 6: run a bounded backfill for legacy data

The backfill was a dormant Job template, not an always-on Deployment. An operator activated it only after read-write traffic was stable.

The job was:

  • bounded: batch size and total rows per run had hard positive limits;
  • idempotent: already encrypted rows were not selected;
  • restartable: a later run continued from remaining legacy rows;
  • concurrency-safe: each update compared the legacy values it originally read;
  • quiet: output contained counts, never answer content;
  • fail-closed: no KMS meant no plaintext-clearing update.

The essential update pattern was:

UPDATE response
SET plaintext_value   = NULL,
    encryption_version = 1,
    encrypted_data_key = :encryptedDataKey,
    value_ciphertext   = :ciphertext,
    value_iv           = :iv,
    value_auth_tag     = :tag
WHERE id = :id
  AND encryption_version IS NULL
  AND plaintext_value IS NOT DISTINCT FROM :valueOriginallyRead;

If a user or operator changed the answer after the job read it, the predicate affected zero rows. The job did not overwrite the newer value. A later batch encrypted the current representation.

The total limit applied across all response and revision tables, not separately to each one. This prevented an apparent “5,000 row run” from unexpectedly changing 15,000 rows across three tables.

Step 7: measure progress without exposing content

Backfill observability used counts and state transitions:

legacy rows remaining
encrypted rows by version
rows updated in this run
optimistic comparisons skipped
KMS operations and latency
decrypt failures by stable category
cleanup candidates and completions

Dashboards never displayed answers, ciphertext, encrypted keys, session cookies, or request bodies. Encryption context was treated as non-secret but still kept free of personal data because KMS audit systems record it.

The completion gate was not merely “Job succeeded.” It required zero eligible legacy rows, successful sampling through normal reads, constraints still valid, backup verified, and no unexplained decrypt errors.

Step 8: define rollback by phase

Rollback meaning changed as the rollout progressed.

Before encrypted writes

Application and schema rollback were conventional. The new columns contained no unique data.

After read-only validation

The application could return to disabled only if controlled encrypted test rows were removed or converted first. Otherwise old code would not read them.

After read-write activation

Turning write mode off stopped new encryption but did not make existing encrypted rows readable by old code. Compatible readers had to remain deployed.

After plaintext-clearing backfill

Dropping envelope columns was destructive. Recovery required an explicit decrypt-to-legacy migration while the keys were available, followed by verification and only then a schema rollback.

This is why “rollback” was written as a decision tree rather than one command.

Step 9: use GitOps as an approval boundary

The active Kustomize render stayed in disabled mode while code and schema changes were reviewed. Secret examples and backfill Jobs remained outside the active resource graph.

A preflight rendered frontend, backend, and runtime overlays and asserted:

  • the expected Secret references existed in the workload contract;
  • no example or placeholder secret leaked into the active render;
  • the refresh-cookie contract matched between components;
  • phase-one encryption mode remained disabled;
  • the backfill Job was dormant;
  • image references were immutable digests during promotion.

Git merge alone did not activate encryption. Promotion, Secret provisioning, migration completion, mode change, and backfill were separate reviewed events. That separation made the deployment legible.

Step 10: close temporary access paths after verification

Local verification often needs temporary connectivity to a private service. It is important to distinguish three cases:

  • Direct in-cluster or node-local service access: no tunnel exists; stopping the local client or dev server ends the traffic.
  • `kubectl port-forward`: a local listening process exists and must be terminated explicitly.
  • SSH or VPN tunnel: a persistent process or session may remain and must be closed and verified.

In this implementation, local event-page verification used a development server making ordinary outbound requests to a private ClusterIP reachable from the node. There was no kubectl port-forward, SSH tunnel, proxy process, or cluster route created. The only long-running process was the local frontend server, bound to 127.0.0.1:3000 so it was not exposed on the public interface.

The closing check was simple:

ss -ltnp
  -> 127.0.0.1:3000 only

no kubectl port-forward process
no ssh -L / ssh -R process
no temporary Service or Ingress

When local review is finished, stopping that development server ends its private backend traffic. There is no separate internal k3s connection to tear down.

The complete rollout sequence

The final order was intentionally boring:

  1. Review the threat model, restore procedure, and retention scope.
  2. Apply backward-compatible Liquibase columns and constraints.
  3. Deploy compatible code with encryption disabled.
  4. Provision BFF and KMS secrets outside Git.
  5. Verify workload identity and KMS context policy.
  6. Enable read-only compatibility and run synthetic canaries.
  7. Confirm every reader and worker supports encrypted rows.
  8. Enable read-write and watch the first real envelopes.
  9. Run bounded, idempotent backfill batches.
  10. Prove zero eligible legacy rows remain.
  11. Rotate keys through the keyring and stored key identifiers.
  12. Keep compatible readers until retention removes the last historical envelope.

No phase relied on hope, and no failure path silently returned to plaintext.

What I would preserve in the next system

The reusable lesson is not a particular cloud SDK or YAML layout. It is the separation of boundaries:

  • UX explains collection.
  • Server validation bounds collection.
  • Authorization binds data to its owner.
  • BFF architecture reduces browser credential exposure.
  • Envelope encryption reduces database plaintext exposure.
  • Retention removes every representation.
  • Compatibility modes make migration reversible for as long as possible.
  • GitOps gates turn security assumptions into reviewable state.

If one of these controls becomes “the security solution,” the design is probably incomplete. Together they reduce different failure modes without asking one layer to do a job it cannot do.

Return to the beginning of the series: Privacy-First Event Forms.

This article intentionally omits organization names, domains, credentials, infrastructure addresses, and identifying production details. The architecture and lessons remain real.