A backend developer’s path into platform engineering
I moved from owning application code to owning the path that keeps it alive
Backend development taught me contracts, transactions, migrations, failure handling, and application performance. DevOps forced me to follow those concerns beyond the process boundary: into CI verification, image construction, GitOps promotion, scheduling, routing, persistent state, telemetry, and recovery. The first version only ran containers. The final operating model could explain what was deployed, why traffic reached it, how capacity was reserved, and how an incident would be diagnosed or reversed.
This is not a polished architecture invented after the fact. Each control came from a real symptom: Pending Pods on an idle-looking node, a non-root CrashLoopBackOff, an eight-minute cache upload, conflicting promotions, incomplete traffic evidence, or a stale controller view. I turned each failure into a repeatable check, policy, alert, test, or runbook.
01 · The brief
The project was an operating model, not a cluster installation
Installing K3s gave me a Kubernetes API, a scheduler, containerd, networking, and controllers in a compact distribution. It did not give me a delivery policy, a backup guarantee, meaningful dashboards, or an incident process. Those had to be designed around the application.
The engineer behind the platform
My contribution was not installing tools; it was connecting their responsibilities
I worked across application code, CI, images, GitOps, Kubernetes, traffic, state, security, and telemetry. When a boundary failed, I followed the evidence into the owning layer, corrected the contract, and documented the reasoning so the same problem would be easier for the next developer to recognize.
02 · Backend developer to platform operator
The difficult part was not YAML; it was expanding the boundary of responsibility
As a backend developer, I was used to owning behavior inside the application: request validation, transactions, schema changes, cache behavior, background work, and error handling. The move into DevOps began when I stopped treating deployment as the moment after development and started treating runtime behavior as part of the software design.
If the application passes its tests, the release is ready.
The image, runtime user, configuration, probes, route, capacity, and database migration can still fail independently.
A release is a chain of contracts, not a compiled artifact.A process is either running or stopped.
Kubernetes manages desired replicas through Deployments and ReplicaSets; a running Pod may still be unready, and a deleted Pod may be normal reconciliation.
Diagnose from controller intent to workload state, not from one process upward.Low CPU usage means the server has room.
The scheduler uses declared requests, while cgroups enforce runtime limits and memory pressure can terminate a container.
Capacity is both an accounting model and a runtime behavior model.An API URL points to the service I need.
DNS, TLS, ingress, Gateway, HTTPRoute, Service, endpoints, sidecars, and NetworkPolicy each own a different part of connectivity.
Follow packets by boundary; do not collapse every failure into ‘the network.’Logs tell me what the application did.
A production question may require scheduler events, metrics, mesh topology, a distributed trace, logs, image metadata, and Git state together.
Observability is correlation across evidence, not a larger log viewer.A database backup means the data is safe.
Storage attachment, credentials, schema version, restore ordering, off-site copies, and an actual recovery attempt determine whether the backup is useful.
Durability is demonstrated by restore, not declared by backup.The largest change in my thinking was this: application correctness ends at no single boundary. Code, image, configuration, scheduler, network, storage, and recovery all participate in the user-visible result.
Where the learning became real
I did not learn the platform in a straight line
I learned it by being wrong in increasingly useful ways. The important part was not avoiding every mistake; it was following each symptom far enough to understand which boundary I had misunderstood, then leaving a better control behind.
The first surprise: an idle server could still reject a Pod
I was looking at roughly twenty percent CPU usage and asking why Kubernetes refused to start another workload. My backend instinct said there was plenty of machine left. The scheduler was answering a different question: how much capacity had already been promised through requests? That incident was the moment resource configuration stopped looking like deployment decoration and became part of application design.
Now I read usage, requests, limits, allocatable capacity, rollout surge, and sidecar overhead together before changing a number.
The second surprise: a green pipeline could still leave users on the old release
The application had merged and the image build had succeeded, yet the environment had not changed. The missing step was not inside the application repository at all: two promotion branches were competing to update the same GitOps manifest. Until then I had treated deployment as the last stage of CI. I learned that delivery crosses repository and controller boundaries, each with its own state and failure modes.
Now I trace commit, pipeline, digest, promotion, Argo CD revision, ReplicaSet, and running Pod as one release chain.
The third surprise: more dashboards did not automatically create more understanding
When Kiali showed an unknown edge, my first reaction was to remove the unknown. That was the wrong question. The edge could represent traffic without mesh identity, a health check, or passthrough communication. I had to learn what each tool could and could not prove before its visualization became useful evidence.
Now I use topology to choose the next question, traces to follow one request, logs to inspect events, metrics to see trends, and Kubernetes objects to understand controller decisions.
The lasting lesson: operating software means preserving its story
The platform became understandable only when I could explain not just what was running, but how it got there and what would happen next: which source revision produced the image, why the scheduler accepted the Pod, how traffic selected it, where state was written, and which evidence would remain if it failed.
That is the standard I now bring back to backend work: every important behavior needs an owner, a boundary, observable evidence, and a recovery path.
03 · Concepts before products
I had to understand the system before I could choose its tools
My early mistake was learning product names before learning the problems they represented. Kubernetes vocabulary became useful only after I could answer more basic questions: Who accepts a connection? How is a destination selected? What capacity has already been promised? Which process may be replaced? Which data must survive? What evidence would prove the diagnosis?
An HTTPRoute, for example, is not “the thing that makes HTTP work.” It is a declarative routing decision between an attached Gateway and a Service: match this hostname and path, then forward to this backend. Resource management is similarly not “giving a Pod some CPU.” It is the discipline of balancing scheduling guarantees, burst ceilings, failure behavior, and rollout headroom on finite nodes.
Capacity is a promise, not a usage graph
CPU and memory usage describe the past. Requests reserve scheduling capacity; limits bound runtime behavior; allocatable capacity is what the node can offer after system overhead.
An address is not a route
DNS resolves a hostname, a Gateway accepts traffic, an HTTPRoute matches host and path, a Service gives changing Pods a stable identity, and endpoint readiness decides which Pods may receive traffic.
Desired state and live state are different truths
Git records the reviewed intention. Argo CD compares it with the cluster. Kubernetes controllers keep reconciling Deployments, ReplicaSets, and Pods toward that intention.
Replaceable compute and durable state need different rules
Frontend and backend Pods should be disposable. Database files, queues, and recovery evidence are not. A PVC preserves storage attachment; a backup creates another copy; only a restore test proves recoverability.
A signal is evidence, not yet a cause
Metrics reveal trends, logs record events, traces connect spans in one request, Kubernetes events explain controller and scheduler decisions, and topology shows service relationships.
04 · The architecture
The current system in five layers
The most useful mental model is not a list of tools. It is a set of boundaries. Each boundary owns a different question, and not every component sits in the live request path.
The public boundary
DNS and proxying resolve the hostname, the WAF filters obvious abuse, TLS protects the connection, and the VPS firewall exposes only the intended entry ports.
- DNS
- CDN / WAF
- TLS
- Firewall
The cluster entry path
Istio ingress receives the connection. Gateway API routes by host and path, Services provide stable discovery, and Envoy sidecars produce mesh telemetry around application traffic.
- Istio
- Envoy
- Gateway API
- Service
The application workloads
Deployments own stateless Pods, probes decide readiness, ReplicaSets enable rolling updates, and requests and limits shape what the scheduler may place on the node.
- Deployment
- Pod
- ReplicaSet
- Probe
State and asynchronous work
Redis serves short-lived data, PgBouncer protects PostgreSQL connection capacity, Kafka carries durable events, and workers consume outbox records without blocking the request path.
- Redis
- PgBouncer
- PostgreSQL
- Kafka
The operating surface
Metrics, logs, traces, mesh traffic, cluster objects, and eBPF signals answer different questions. No single dashboard is treated as the whole truth.
- Grafana
- Loki / Tempo
- Kiali
- Headlamp / Coroot
Architecture decision record
The choices are only credible when their costs are visible
Start with K3s, keep the Kubernetes contract
- Decision
- I used K3s because one 8-vCPU, 32-GiB VPS did not justify a multi-machine control plane.
- Reason
- It preserved the API, scheduler, controllers, containerd runtime, manifests, and operational vocabulary I needed to learn.
- Cost accepted
- The platform is not highly available. Node failure remains a service outage, so recovery evidence matters more than pretending one node is resilient.
Promote immutable digests through GitOps
- Decision
- Application pipelines build images; a separate deployment repository records the digest that should run.
- Reason
- This creates a reviewed chain from commit and pipeline to manifest revision, ReplicaSet, and Pod image.
- Cost accepted
- Concurrent promotions can conflict in the same manifest, so promotion ordering and latest-release ownership need an explicit contract.
Adopt Istio progressively, not with a blind takeover
- Decision
- I observed the live control plane first, added render validation, then moved ownership under GitOps.
- Reason
- Traffic identity, mTLS, and mesh telemetry were valuable, but replacing an existing control plane in one step created unnecessary blast radius.
- Cost accepted
- Every sidecar consumes resources and every mesh policy adds another failure boundary; capacity and policy exceptions must be budgeted deliberately.
Treat recovery as a continuously tested feature
- Decision
- Backups are copied off the workload path and scheduled restore checks produce visible evidence.
- Reason
- A successful backup command proves that bytes were written, not that the database can be recovered when needed.
- Cost accepted
- Restore tests consume storage, compute, credentials, and maintenance attention, but remove the larger risk of discovering an unusable backup during an outage.
05 · Commit to runtime
A release is two repositories and several explicit gates
Application code and desired cluster state are intentionally separated. The application pipeline proves the code and produces an immutable image. A promotion changes the deployment repository. Argo CD then reconciles that reviewed state into K3s.
The distinction that changed how I debug pipelines: GitLab cache, job artifacts, Kaniko layer cache, the container registry, and a Maven repository such as Nexus solve different problems. Treating them as one cache makes build time and failure ownership impossible to reason about.
What I implemented
The platform is a set of versioned operating controls, not a tool screenshot
06 · Browser to database and back
The request path I can now narrate without skipping a boundary
07 · Failure stories
The mistakes that built the operating model
These incidents are more representative of the work than a clean architecture screenshot. Each one changed a manifest, a pipeline, a dashboard, or a runbook.
Scheduler: Insufficient cpu
The node looked idle, but new Pods stayed Pending
- What I got wrong
- I initially read node CPU usage as available scheduling capacity. Kubernetes does not place a new Pod from live usage; it checks the sum of declared CPU requests.
- What changed
- I audited requests across workloads, reduced inflated reservations, kept bounded limits for burst capacity, and introduced a 75–80% request-headroom target for rolling updates and CI jobs.
- Rule I keep now
- Usage, request, limit, and allocatable capacity are four different numbers. Capacity planning starts with all four.
Permission denied in the runtime filesystem
A secure non-root image entered CrashLoopBackOff
- What I got wrong
- The Pod security context correctly forced an unprivileged user, but the image still assumed that its web server could write to root-owned cache directories.
- What changed
- I replaced the root-dependent runtime with a small non-root static server, preserved health and readiness endpoints, and verified the same UID contract in the image and Deployment.
- Rule I keep now
- A securityContext cannot make an image rootless by declaration. The image filesystem and the process must support the same contract.
Kaniko spent most of the build snapshotting node_modules
A tiny frontend change produced an eight-minute cache push
- What I got wrong
- The Dockerfile copied the complete package manifest before dependency installation. Changing a script invalidated the dependency layer even though the lockfile and dependencies were unchanged.
- What changed
- I separated dependency metadata from scripts, guarded synchronization with a test, and kept the dependency layer stable unless dependencies genuinely changed.
- Rule I keep now
- A cache exists only if layer boundaries match the rate of change. Cache hit ratio is an architecture decision, not a checkbox.
The GitOps merge request conflicted in one Deployment manifest
Two successful builds produced a blocked promotion
- What I got wrong
- Multiple promotions changed the same image digest and build metadata lines. The application repository was healthy, but the deployment repository had concurrent desired-state updates.
- What changed
- I rebased the latest promotion onto deployment main, preserved the newest digest, validated the rendered YAML, and documented a latest-promotion-wins workflow.
- Rule I keep now
- Build success and deployment success are separate states. A promotion queue needs its own concurrency model.
Kiali reported disabled metrics or showed unknown edges
The traffic graph existed, but it could not explain a request
- What I got wrong
- I expected a service-mesh graph to replace logs and traces. Kiali also depended on a correct Prometheus endpoint and consistent workload labels.
- What changed
- I fixed the metrics integration, added a repeatable traffic probe, connected Tempo, and split the workflow: Kiali for topology, Tempo for a request, Loki for events, and Grafana for trends.
- Rule I keep now
- Topology is not tracing. Unknown can mean external traffic, passthrough traffic, or missing identity; it is a classification clue, not automatically an outage.
Resource not found in cluster
A deleted Pod still looked broken in Argo CD
- What I got wrong
- During a normal rolling update I kept an old Pod detail view open. The new ReplicaSet was healthy, but the UI still pointed at the deleted object.
- What changed
- I checked the Deployment, current ReplicaSet, replacement Pod, events, image digest, and application health before refreshing the stale resource view.
- Rule I keep now
- Controllers own desired state; Pods are replaceable instances. Diagnose from the owner downward, not from a stale leaf upward.
08 · How I operate it now
One incident, several views, one evidence chain
I no longer open dashboards at random. I start with the question, choose the evidence type, and keep the image digest, namespace, workload, time window, and trace ID aligned between tools.
What I teach other developers
I explain operations as a causal story, not a vocabulary test
The documentation that grew around this platform is aimed at developers crossing into operations. I do not begin with “install Istio” or “write a Deployment.” I begin with the request, the resource promise, the state transition, or the failure that creates the need for those tools.
Start with the journey, not the YAML
I explain a browser request from DNS and TLS through ingress, route, Service, ready Pod, database, response, and telemetry before introducing the manifests that implement each boundary.
Teach the differences that prevent incidents
Usage is not a request. A Service is not a Pod. A successful pipeline is not a deployment. Topology is not a trace. A backup is not a proven restore.
Use the failure as the lesson
A Pending Pod teaches scheduler promises; a CrashLoop teaches image and runtime contracts; a promotion conflict teaches that delivery has concurrency, not just automation.
Leave the next operator a decision path
I write visual runbooks that move from symptom to owner, evidence, reversible test, correction, verification, and rollback instead of leaving a list of commands without context.
My goal is not to make infrastructure look simple. It is to make complexity navigable: show the boundaries, name the tradeoffs, preserve the evidence, and give the next person a safe first move.
Security and state
Security became a chain of boundaries, not a hardened checkbox
cert-manager automates certificate issuance and renewal; Istio mTLS protects workload-to-workload identity inside the mesh; Tailscale exposes operational tools privately; and Sealed Secrets lets encrypted secret declarations live in Git. They solve adjacent problems, but none replaces the others.
09 · How I would start the next system
The questions now come before the tools
Platform vision
The next step is not more tools; it is a clearer contract between developers and operations
What the project demonstrates
Production thinking is the ability to preserve causality
I can now connect a source change to a pipeline, an image digest, a GitOps revision, a ReplicaSet, a Pod, a request, a database interaction, an alert, and a recovery action. More importantly, I can explain why each boundary exists, what evidence it produces, and how it fails. The platform is still intentionally compact and single-node; I do not present it as high availability. I present it as evidence that I can reason honestly about capacity, routing, state, delivery, security, observability, and recovery, then carry those decisions into a larger Kubernetes environment.
If I were speaking to the backend developer I was at the beginning, I would not tell him to memorize Kubernetes objects. I would tell him to keep following the behavior he already cares about. Follow the request beyond the controller, the transaction beyond the database commit, the release beyond the pipeline, and the failure beyond the first log line. DevOps became understandable when I stopped seeing it as a separate world and started seeing it as the rest of the software system.