Skip to content

Blog

Speeding Up CI: The 'Detect' Job Pattern

I still recall the sting of watching our build pipeline grind through a costly set of checks. It burned minutes of compute. Then it just said: "No relevant changes found. Nothing to audit." It was for a small doc change, a quick typo fix. Yet our system ran a full identity check, pulled private reviewer data, and spun up a Go setup. All that, to confirm its helm chart linter had nothing to do. Multiply that across dozens of pull requests a day, and the hidden cost of wasted CI work adds up fast. That moment pushed us to rethink how we gated our priciest checks.

Operator RBAC from Code Markers

I still remember the late-night debugging session. I was staring at a permission denied error in our Kubernetes integration testing operator logs. We'd just introduced a new custom resource. Despite our best efforts, the operator couldn't get the permissions it needed to update the status field. We meticulously crafted the RBAC YAML. We tried to grant just enough access, but the Kubernetes API kept rejecting our requests. This was a familiar frustration, common in every operator project I'd touched. The manual process of defining roles, role bindings, and service accounts, then syncing them, felt like a constant uphill battle. It ate into valuable development time.

The Monday Morning PR Wall

One Monday morning, I stared at a GitHub dashboard overflowing with 70+ open pull requests. Each one, a dependency update generated by our diligent bot, sat there, waiting. Manually triaging these across a fleet of microservices was not just tedious; it was a significant drain on our team's focus and an ever-present source of operational anxiety. We were constantly asking ourselves: Which ones are critical? Which can wait? Do these even apply to our core platform components or just auxiliary tools?

The Cold Sweat Deploy

I remember the cold sweat, staring at a failing deployment log, knowing that a single, forgotten manual step was costing us precious minutes of critical service downtime. That was the moment I truly understood the chaos of non-declarative operations, and the urgent need for a better way to manage our applications across our growing fleet of cluster environments.

The Last Service Account Key

$ git log --all --oneline -- '**/service-account.json' | wc -l
47

$ git log --all --oneline -- '**/service-account.json' | head -1
a3f8c2e delete: remove production service account key

That commit sits in your history like a monument. Not because of what it added, but because of what it finally took away. Forty-seven commits that existed only to move secrets around, rotate them, revoke them, apologize for them, and eventually eliminate them.

That last deletion was the sound of the door closing on an entire class of infrastructure vulnerability.

The Architecture That Couldn't Be Breached

Container escape achieved. Attacker privilege: still none. Why?

The breach happened. The forensics confirmed it. Shellcode executed inside the container. Root user, full system access, network connectivity. All compromised. Everything the attacker needed to pivot was there.

None of it worked.

The escaped container had no network access to other services. Secrets were never mounted into the pod. The attacker had no credential to steal. The host firewall blocked outbound connections. The network policy denied access to the control plane. The RBAC denied any service account permissions.

The container was compromised. The architecture was not.

This is what defense in depth looks like when it actually works.