io.github.CloudCorpRecords/testdelta

testdelta

Diff two named test result sets and report exactly which tests regressed, which were fixed, which were added, and which were removed. Never raw counts without names: every category in the human report lists the test names in it.

What it does

testdelta compares a base result set against a head result set. Each set maps a test name to one of three statuses: pass, fail, or skip. From the two sets it computes:

• Regressions: tests that were pass or skip in base and are fail in head.
• Fixes: tests that were fail in base and are pass in head.
• Added: tests that exist only in head, with their head status.
• Removed: tests that exist only in base, with their base status.
• Still-failing: tests that are fail in both sets.
• Unchanged: tests present in both sets with the same status, other than fail in both.
• Other changes: tests present in both sets whose status changed in a way that is neither a regression nor a fix, for example pass to skip.

Name lists are sorted alphabetically within each category, and the output is deterministic: no timestamps and no random ordering.

Findings highlight the categories a human should look at first:

• One high finding per regression (rule REGRESSION, evidence such as base=pass head=fail).
• One info finding per fix (rule FIXED).
• One medium finding per removed test that was passing (rule REMOVED-PASSING).
• One low finding per added test that is failing (rule ADDED-FAILING).

Input formats

Directory with base.json and head.json

Point testdelta at a directory containing two files:

base.json:

{
  "test-login": "pass",
  "test-checkout": "pass",
  "test-search": "fail"
}

head.json:

{
  "test-login": "fail",
  "test-checkout": "pass",
  "test-search": "pass"
}

In this example test-login is a regression and test-search is a fix.

Single file with base and head objects

A single JSON file can carry both sets:

{
  "base": {
    "test-login": "pass",
    "test-search": "fail"
  },
  "head": {
    "test-login": "fail",
    "test-search": "pass"
  }
}

Missing files, missing base or head objects, malformed JSON, and status values other than pass, fail, or skip are errors. The CLI reports the error and exits with code 2.

Install

npx @reneturcios/testdelta diff ./results

Or install it globally:

npm install -g @reneturcios/testdelta

CLI usage

# Human readable report (default target is the current directory)
testdelta diff [path]

# Machine readable JSON report
testdelta diff [path] --json

# Run as an MCP server over stdio
testdelta mcp

# Print the version
testdelta --version

Exit codes:

• 0: the verdict is NO-REGRESSION.
• 1: the verdict is REGRESSED.
• 2: usage error, including missing or invalid input.
• 141: the reader closed the output pipe early (the shell-visible equivalent of SIGPIPE).

In the human report, each category lists test names, capped at 50 names per category. When a category has more, the list ends with an explicit and N more line instead of truncating silently. The JSON report always carries the full findings list.

MCP tool

The MCP server exposes one tool:

diff_test_results

• Input: path, the target directory or single result file.
• Output: the full JSON report, with the verdict, the counts for every category, the findings, and a summary that names the counts and the first regression names.

Verdict rules

• REGRESSED: at least one regression exists, meaning a test that was pass or skip in base is fail in head.
• NO-REGRESSION: no regressions. Added failing tests, removed tests, and tests that were already failing in base do not change the verdict on their own; they are reported as findings and counts so a human can decide.

Limits

• testdelta works on result files. It does not run tests, and it does not produce the pass, fail, or skip values itself; a test runner or CI job has to write the result sets first.
• Test name matching is exact string equality. A renamed test appears as one removed test and one added test, not as a change.
• Only the three statuses pass, fail, and skip are understood. Map any other runner outcome onto one of them before diffing.