Skip to content

Blog

The GKE Cluster That Nobody Could Break

Day 1 of pentest. Security firm arrives with methodology, tools, and confidence. The plan is simple: find gaps in the Kubernetes cluster, prove impact, deliver a detailed report of findings.

Day 2. They're quiet. Too quiet.

Day 3. Meeting request. Not the kind where they show you their findings.

"We found nothing. Well, nothing critical. Actually, we found nothing at all. This is the best-hardened cluster we've tested. Want to know what you did right?"

That's not how pentest reports usually end.

The 3am Incident That Followed The Playbook

3:17am. The pager vibrates on the nightstand. Half asleep, hand fumbles for phone. The message is three lines. Pod restart storms. API latency spiking. Customers seeing timeouts.

The engineer's first thought isn't "oh god, what now." It's automatic: "open the runbook."

Muscle memory takes over. Hands pull up a laptop still warm from yesterday. The playbook is right there: decision tree, diagnostic steps, escalation paths. No thinking required. Just follow the checklist.

Twenty-three minutes later, the incident is closed. Every step documented. The postmortem writes itself.

This is what happens when you stop improvising and start automating response.

The Policy That Wrote Itself

12 teams. 47 namespaces. 1 security requirement. 0 teams wanted to write policies.

The mandate came down: all workloads need pod security policies. No root containers. No privileged escalation. No host volumes. Standard stuff. Every team got the requirement. Then the work stalled.

Policy-as-Code is powerful. Enforcement at admission time stops bad deployments before they reach etcd. But power has a price: someone has to write YAML.

Team A wrote a policy. 34 lines. Solid.

Team B copy-pasted it. Forgot to update the label selectors. Now it applies to everything, including system services. Everything gets rejected. Team B spends four hours debugging why their monitoring won't deploy.

Team C started from scratch. Different syntax. Nested conditions. Hard to read. Works, mostly.

Team D went with "we'll do it next sprint." Still waiting.

The pattern was obvious: enforcement is easy. Enforcement at scale isn't. Every team writing their own policies means every team makes the same mistakes.

Same mistakes repeated 12 times is an incident waiting to happen.

The Checklist That Passed the Audit

Audit notice: 30 days. Evidence requested: everything.

Two weeks of scrambling. Teams pulling logs. Spreadsheets cross-checking commits. Patch requests hunting for proof that code reviews actually happened. Documentation written in panic mode. Governance questions without answers. A process that lived in people's heads, not in tooling.

Then one team showed their checklist. One list. One enforcement mechanism. Every claim tied to evidence collected automatically.

Audit was over in 2 weeks instead of 6.

The CLI That Replaced 47 Shell Scripts

47 shell scripts. 12 CronJobs. Zero test coverage. One production incident that forced a rebuild.

The kubectl plugin pattern couldn't handle our complexity. Shell scripts worked until they didn't. No type safety. No testing. Debugging meant reading logs after failures.

Then came the rewrite: One CLI. 89% test coverage. Runs in CronJobs and Argo Workflows. Distroless container. Multi-arch binaries. Production deployments via Helm.