47 shell scripts. 12 CronJobs. Zero test coverage. One production incident that forced a rebuild.
The kubectl plugin pattern couldn't handle our complexity. Shell scripts worked until they didn't. No type safety. No testing. Debugging meant reading logs after failures.
Then came the rewrite: One CLI. 89% test coverage. Runs in CronJobs and Argo Workflows. Distroless container. Multi-arch binaries. Production deployments via Helm.
The deployment had no resource limits. Memory leaked. The node crashed. Three pods died.
The developer missed the requirement. Code review missed it. CI passed. Tests skip resource checks.
Admission control stops this. Before etcd. Before production. Before damage.
The email arrived on a Monday. "SOC 2 audit in 30 days. Need evidence of secure development practices."
We had policies. Documentation. Training slides. None of it mattered.
Auditors don't want to hear what you say you do. They want to see what the system forces you to do.
My workflow reads deployment mappings from a ConfigMap mounted as a volume. Five milliseconds, zero API calls. But what happens when the mount fails?
The answer isn't "return an error." It's "try the next thing."
The cluster scan was killing performance. Every image push meant querying the API server for all deployments. 5-10 seconds of latency. Hundreds of API calls per day asking the same question.
The answer changed maybe once a week, but I was asking every five minutes.
This is the pattern that fixed it.
A few days ago I wrote about why release-please PRs don't trigger builds and proposed a dual-trigger pattern as the fix. Today I discovered that pattern is a workaround with side effects. Here's the actual solution.
The workflow looked perfect. CONTRIBUTING.md in our central repository, automatically distributed to all 75 repositories. Any change triggers PRs across the organization.
Then release-please bumped the version from 1.4.1 to 1.4.2.
I wanted to version CONTRIBUTING.md independently from the main project. Separate changelog. Separate release cycle. Separate tags.
Release-please said no.
Every time someone pushed a container image, my Kubernetes API server winced. The workflow that was supposed to be "instant" took 5-10 seconds and hammered the cluster with requests.
This is the story of how I turned that into 5 milliseconds with zero API calls.
Update: There's a Better Way
The dual-trigger pattern described below is a workaround, not the real fix. See The Real Fix for Release-Please Triggers for the proper solution using GitHub App tokens.
The release-please PR looked perfect. Clean changelog. Proper version bump. Ready to merge.
One problem: the build pipeline never ran. Branch protection blocked the merge. No required checks had passed, because no checks had started.
This is the story of a GitHub Actions limitation that wastes hours of debugging time, and the pattern that fixes it in two lines of YAML.