What AI Could Not Infer from the Error Message

01 · The debugging gap

What AI could not infer from the error message

AI can search a codebase, compare logs, and produce plausible hypotheses quickly. That is useful. But a plausible explanation is not the same as a production diagnosis. The hardest failures I worked through were not hidden behind obscure syntax. They lived between boundaries: a database version and an API DTO, a pipeline and a GitOps promotion, a usage graph and scheduler accounting, or a security response and a service graph.

The tool could suggest where to look. The engineering work was deciding what each signal could prove, finding the missing contract, and verifying that the correction preserved data and behavior.

SignalA request failed

HypothesesSeveral causes fit

EvidenceEliminate causes

ContractFind the broken boundary

ControlEncode the fix

02 · The incident

The backend rejected publish after four questions were added

A draft business event started at lockVersion = 1. Four form questions were added through separate mutations. Each successful mutation changed the same aggregate and incremented its optimistic-lock version. The record was now version 5.

Create draftv1

Add question 1v2

Add question 2v3

Add question 3v4

Add question 4v5

The demo workflow expected the current version to come back from the form-create or admin response. That DTO did not expose lockVersion. The subsequent publish command therefore sent an empty or stale expected version. The backend rejected it with a 400 response.

What the response suggested

Something about the submitted event was invalid

Migration, form payload, validation, and partial writes were all credible first hypotheses.

What the system actually did

It prevented a stale write from overwriting newer state

The event stayed safely in DRAFT, all questions remained stored, and no duplicate was created.

UPDATE business_event
SET status = 'PUBLISHED',
    lock_version = lock_version + 1
WHERE id = :id
  AND status = 'DRAFT'
  AND lock_version = :expectedVersion;

When this update affects zero rows, the database is not explaining why. The record may not exist, its status may have changed, another mutation may have increased the version, or the client may have omitted the token. That ambiguity is exactly where evidence must replace guessing.

03 · The investigation

I stopped reading the 400 as the root cause

QuestionObserved evidenceAnswer

Did the migration fail?Migration history was complete and the schema was usable.No

Was the form partially written?The draft and all four questions existed; no duplicate row appeared.No

Did another mutation change the aggregate?The database held lockVersion 5 after four successful child mutations.Yes

Did publish use the current version?The request carried an empty or stale value because the admin response omitted it.No

400 responseDraft and children intactDatabase version = 5Publish version empty or staleAdmin response contract omitted the concurrency token

This was not a database-only problem and not a frontend-only problem. Optimistic locking crossed the database column, persistence mapping, application command, response DTO, client state, and publish request. The failure appeared only when the full workflow was exercised.

04 · Recovery and correction

Recover the record first, then repair the contract

Immediate, controlled recovery

Read version 5 and publish explicitly against version 5

This completed the current workflow without disabling the protection. It solved the blocked release, but it did not pretend the admin client was already correct.

Permanent engineering correction

Make concurrency part of the end-to-end API contract

Return the current version from create, read, and every successful mutation response.
Update frontend state atomically from the mutation response instead of guessing the next version.
Require expectedVersion on publish and reject a missing token before business execution.
Model stale writes as a conflict, ideally 409 or 412 when compatibility permits.
Never blindly retry a stale business command; reload state and re-evaluate the invariant.
Add contract, UI, concurrency, negative, and audit tests around the complete workflow.

ContractEvery mutation returns the new version.

UI flowFour mutations followed by publish use version 5.

ConcurrencyTwo admins publish; only one stale command is accepted.

NegativeMissing and stale tokens return a machine-readable conflict.

AuditRejected commands do not change state or create duplicates.

05 · The recurring pattern

Four other failures that lived between tools

The node looked mostly idle, but a new Pod stayed Pending.

First guess: The scheduler or cluster was broken.

Hidden boundary: Runtime usage and schedulable requests answer different capacity questions.

Decisive proof: Pod events reported insufficient requested CPU, not high observed CPU usage.

The pipeline was green, but users still received the old release.

First guess: The application image had not been built correctly.

Hidden boundary: Build success did not prove GitOps promotion, Argo revision, or live image identity.

Decisive proof: The commit -> digest -> manifest -> ReplicaSet -> Pod chain stopped at promotion.

The service graph turned orange and showed an unknown source.

First guess: A workload or mesh route was unhealthy.

Hidden boundary: External scanner traffic had no workload identity and was being denied at ingress.

Decisive proof: Envoy access logs showed policy 403 responses before an application Pod was selected.

Generating a large PDF caused memory pressure inside the application Pod.

First guess: The HTTP response itself was too slow.

Hidden boundary: A byte-array workflow retained the whole document in heap before transmission.

Decisive proof: Streaming or bounded temporary-file handling moved the pressure away from unbounded heap growth.

06 · The reusable playbook

Use AI for breadth; keep proof and decisions owned by the engineer

1. Name the symptomRecord the exact response, time, actor, operation, and affected state.

2. Draw the boundariesList browser, API, queue, database, CI, GitOps, cluster, and proxy ownership only where relevant.

3. Ask what each signal provesA 400, green pipeline, Running Pod, or orange graph is evidence, never the complete diagnosis.

4. Eliminate hypothesesUse state, version, owner reference, digest, request ID, logs, metrics, and traces to close possibilities.

5. Choose a reversible actionPreserve protection and data while restoring service; do not bypass a guard just because it blocked progress.

6. Encode the lessonRepair the contract, then add tests, telemetry, validation, and a runbook so the same ambiguity cannot return.

The protection was not the bug

Optimistic locking did exactly what it was designed to do. The real defect was that one boundary failed to carry the information required by the next. That is the kind of problem an assistant can help investigate, but only an engineer with system context can close responsibly.