The Queue That Deleted Itself
Eight workflows queued. Seven deleted themselves before execution. Zero wasted builds.
Eight workflows queued. Seven deleted themselves before execution. Zero wasted builds.
The workflows were slowing down. Not dramatically. Just... gradually. Week by week.
Then the logs started showing warnings: API rate limit approaching threshold.
100+ workflows per day. Each workflow making 2-3 Kubernetes API calls to read ConfigMaps. That's 200-300 API requests per day. Not sustainable.
The deployment worked in dev. Unit tests passed. Code review approved. Merged to main.
Production exploded.
The config used a dev secret. Migration locked tables at scale. Feature flag on in staging. Off in prod.
Environmental differences killed us.
The CVE dropped. The container was in production. CRITICAL severity. Remote code execution. Tagged latest.
Nobody knew which deployments used it. Scanning happened post-push. The flaw lived in production six hours.
This is the pattern that stops CVEs before they reach the registry.
The From 5 Seconds to 5 Milliseconds post showed how caching eliminated cluster scans. But that was only half the story. The real magic happens in how events flow through the system.
This is that story.
The deployment had no resource limits. Memory leaked. The node crashed. Three pods died.
The developer missed the requirement. Code review missed it. CI passed. Tests skip resource checks.
Admission control stops this. Before etcd. Before production. Before damage.
The cluster scan was killing performance. Every image push meant querying the API server for all deployments. 5-10 seconds of latency. Hundreds of API calls per day asking the same question.
The answer changed maybe once a week, but I was asking every five minutes.
This is the pattern that fixed it.
Every time someone pushed a container image, my Kubernetes API server winced. The workflow that was supposed to be "instant" took 5-10 seconds and hammered the cluster with requests.
This is the story of how I turned that into 5 milliseconds with zero API calls.