47 shell scripts. 12 CronJobs. Zero test coverage. One production incident that forced a rebuild.
The kubectl plugin pattern couldn't handle our complexity. Shell scripts worked until they didn't. No type safety. No testing. Debugging meant reading logs after failures.
Then came the rewrite: One CLI. 89% test coverage. Runs in CronJobs and Argo Workflows. Distroless container. Multi-arch binaries. Production deployments via Helm.
Eight workflows queued. Seven deleted themselves before execution. Zero wasted builds.
Codecov components were configured. Five components, clean breakdown by package.
Codecov dashboard: "No report uploaded" for all components.
Coverage uploaded successfully. Components showed zero coverage.
The paths looked correct.
secrets: inheritCODECOV_TOKEN worked perfectly in ci.yml.
Called ci.yml from release.yml as a reusable workflow. Codecov failed:
Same token. Same workflow. Different result.
The workflows were slowing down. Not dramatically. Just... gradually. Week by week.
Then the logs started showing warnings: API rate limit approaching threshold.
100+ workflows per day. Each workflow making 2-3 Kubernetes API calls to read ConfigMaps. That's 200-300 API requests per day. Not sustainable.
The provenance file generated perfectly. Build completed. Release uploaded.
slsa-verifier failed:
Same workflow that worked last month. Nothing changed.
Except everything changed.
"What security tools do you use?"
I expected: Snyk. Semgrep. Custom vulnerability scanners. Expensive SaaS subscriptions.
The answer: go test -race.
That's it. That's the security tool.
The OpenSSF Best Practices Passing badge doesn't mandate a specific coverage percentage.
We set our bar at 95% minimum. Above even Gold (90%). Self-imposed. Strategic.
We started at 0%. We wanted the Passing badge. But we knew something important: it's easier to build high standards into a young project than retrofit them later. When we go for Gold, 95% would already be habit.
The workflows had been running for months. Cosign signing. SBOM generation. Release automation.
Everything worked.
Then OpenSSF Scorecard ran: 16 Token-Permissions alerts.
The code scanning tab filled with warnings. All from one workflow file.
The test suite was comprehensive. Table-driven tests. Error paths covered. Race detector enabled.
Coverage: 85.8%. Stuck.
We needed 95%. The team wrote more tests. Coverage stayed at 85%.
The problem wasn't the tests. It was the code.