Work · Code
pr-reviewer.
A pull request goes in. Inline comments land on the lines that need them, and a summary sits at the top.
Companion tool: jira-bridge (posts review findings to Jira)
The problem
A comment on the exact line, naming the bug and a fix.
Imagine a pull request with a one-line change that subtracts the same reserved quantity twice. A reviewer who is already late opens the diff, looks for that kind of mistake, and writes a comment on the line.
The useful result is not a model that “reviews code.” It is a comment on the exact line, naming the bug and a fix, that a person can accept or dismiss.
What we built
Inline comments on the lines they refer to, and one summary.
The pipeline is five steps. A pull request (or a unified diff on disk) is reviewed in short hunks, against a YAML rules file that sets what to check, what to ignore, and how strict to be. Inline comments land on the lines they refer to; comments on lines that are not in the diff are dropped. One summary comment sits at the top. The backend can be a hosted API or an open model on your own machine. An optional companion, jira-bridge, posts those findings to Jira when it is enabled.
A general “this looks risky” comment at the bottom of a pull request is easy to scroll past. A comment on the exact line, naming the bug and a fix, is something a person can accept, edit, or dismiss — and the rules file is how a team says what is worth commenting on.
Results
5 of 5 seeded bugs found.
-
5 / 5
Seeded bugs found
-
Qwen2.5-Coder-7B
Q4_K_M · API backend
-
6 diffs
Five bugs + one ignored
On six diffs — five with a seeded bug, and one documentation-only change that the default rules ignore — the API backend with Qwen2.5-Coder-7B-Instruct (Q4_K_M) found 5 of 5 seeded bugs. The documentation-only diff produced no comments, as intended.
A finding counts as found when a comment is on the same file and side, the line is within the finding’s window, and the comment body contains one of the expected keywords (and does not agree with the bug instead of reporting it).
| Diff | Bug | Path | Expected findings |
|---|---|---|---|
picker-index.diff | off-by-one (index <= len(bins)) | picker.py | 1 |
double-subtract.diff | reserved quantity subtracted twice | stock.py | 1 |
query-concat.diff | SQL built by string formatting | query.py | 1 |
total-returns-count.diff | len(items) where a total was wanted | totals.py | 1 |
discount-formula.diff | price / percent where a fraction was wanted | billing.py | 1 |
docs-only.diff | documentation only (ignored) | — | 0 |
| All | 5 of 5 seeded bugs found | 5 + 0 |
Methodology. Six diffs in samples/diffs, with ground truth in the matching *.truth.json files. API backend, Qwen2.5-Coder-7B-Instruct (Q4_K_M) behind a local llama.cpp server. The raw table is in docs/eval.md. The script is scripts/evaluate.py; the samples are in samples/.
Run it yourself
Three commands, then the demo.
git clone https://github.com/ModelShifts/pr-reviewer
cd pr-reviewer && python -m venv .venv && source .venv/bin/activate
pip install -e ".[api,demo]" && pr-reviewer serveThe 7B figures used a Q4_K_M GGUF through llama.cpp (API backend). On the Hugging Face backend, a CPU-only machine should set MS_HF_MODEL=Qwen/Qwen2.5-Coder-1.5B-Instruct. Python 3.11 or newer.