Developer Tools · Flaky test finder
FlakeLedger
Proof, price and a fix for every flaky test.
Reads the test results your CI already keeps, proves which tests are flaky, prices what they cost, explains why they fail, and stops them blocking merges while real failures still do.
$ flake-ledger analyze ci-runs --repo .
24 tests over 60 runs (29.5 days): 5 flaky (5 proven), 1 broken; about $490 a month
flaky (proven) 15 fails $226 checkout › applies coupon owner @acme/web
passed on retry in 15 runs; UI timing
flaky (proven) 10 fails $151 test_payment_webhook owner @acme/payments
Broken, not flaky: test_csv_export since 4 runs ago
The problem
Everyone knows some tests are flaky, but not which ones for sure, how often, or what they cost. So teams re-run CI until it's green, real failures get waved through as "probably flaky", and the same five tests waste hours every week.
What it does
Reads JUnit XML from pytest, Jest, Vitest, Playwright, Cypress, Go, Maven, Gradle and .NET, fetched straight from past GitHub Actions runs or GitLab pipelines
Proves flakiness: the same test failed and passed on the same commit, or passed on a retry inside one run
Keeps broken tests apart: a test that fails every run since a commit is a real failure and is never hidden
Groups the error messages and names the likely cause with its usual fix: UI timing, timeouts, shared data, network, ports, clocks or runner limits
Prices each flaky test in re-runs and developer time, per month
Writes the flags that quarantine them for pytest, Go, Jest, Playwright or Maven, and one issue per test with its owner from CODEOWNERS
A CI gate that passes when the only failures are known flaky tests and blocks on everything else
Marks each flaky test in the pull request, on the line that defines it, with its evidence and likely fix
How it works
- Fetch the last 100 runs' test results from GitHub Actions
- Get the flaky tests with proof, cost, cause and owner
- Quarantine them, fix the costliest first, and gate CI on real failures
Works with
Runs in your CI
# Weekly flaky-test report
flake-ledger fetch $REPO --workflow tests.yml --artifact 'junit-*' --limit 100
flake-ledger analyze ci-runs --repo . --ledger flake-ledger.json --html flaky.html
# In the test job: pass when only known flaky tests failed
flake-ledger gate test-results/ --ledger flake-ledger.json
Your data
- Runs in your CI or on your laptop; test results are not uploaded.
- Reads GitHub with your own read-only gh login.
- Broken tests are reported, never quarantined.
Who it's for
Questions
Don't retries already handle flaky tests?
Retries hide them and you still pay for every re-run. FlakeLedger measures them, proves which are flaky, and tells you which to fix first.
Could it hide a real failure?
No. A test failing every run since a commit is reported as broken and never quarantined, and the gate only lets known flaky tests through.
Related
Often used with FlakeLedger
RunbookFinder
Engineering knowledge searchAnswers your engineers can trust. Ask how a system works and get the runbook section, the postmortem and the code, with who owns it and whether it's out of date.
Free to start · quote on requestView product → Developer ToolsTestHealer
Self-healing test fixesRedesign without breaking tests. After a redesign, find every UI test selector that broke, the element it meant, and a verified fix as a diff. The same for API version bumps.
Free to start · quote on requestView product → Developer ToolsA11yReady
Accessibility auditAccessible by design, compliant by law. Audit your website against WCAG 2.2 AA and see what each issue means under the European Accessibility Act, the ADA, and UK, Australian and Indian rules.
Quote on requestView product →See FlakeLedger on your own data
A 30-minute call. We run the product on your export, repository or document and walk you through the results.