Skip to content
agentusehub

Find and fix flaky tests in a broken CI pipeline overnight

Claude monitors CI pass rates, identifies worst flaky tests, reproduces them and proposes fixes overnight; pipeline got 20 minutes faster and pass rate rose 35%, with some bad fixes discarded.

Time saved
Not reported
Setup time
Not reported
Running cost
Not reported
Cost range
Not reported

Use case overview

Claude monitors CI pass rates, identifies worst flaky tests, reproduces them and proposes fixes overnight; pipeline got 20 minutes faster and pass rate rose 35%, with some bad fixes discarded.

Reported experience

  1. What they did

    The author gave Claude read access to CI artifacts and had it build tooling to track the rolling 30-day pass/fail rate. It picked the worst flaky tests, hypothesized test versus production causes, and narrowed to minimal repros. Without a deterministic repro, it guessed a fix and re-ran tests overnight to judge it statistically. The author reviewed PRs by day.

  2. How it went

    Over a handful of weeks the pipeline got 20 minutes shorter and the pass rate rose 35%, from roughly 40% on master. Some of Claude's proposed solutions were bad and had to be thrown away.

  3. Worth knowing

    Plan to review every PR yourself; the author discarded some bad fixes, so overnight runs need daytime human checking.

Example prompts

Try with Claude Code
Look at the last [number] CI runs for [repo name] and rank the tests by how often they flake. For the worst [number], reproduce the failure locally, find the cause, and propose a fix on a separate branch. Run each fix [number] times to confirm it's stable, discard any fix that doesn't hold, and summarize what worked. Don't merge anything without showing me the diff first.
Software & DevOps

Port a Chrome extension to Safari

The author had ChatGPT take the Session Buddy extension source from Chrome, port it to Safari and install it, taking about 20 minutes.

ChatGPT