# pr-reviewer

A pull request goes in. Inline comments land on the lines that need them, and a summary sits at the top.

Companion tool: [jira-bridge](https://github.com/ModelShifts/jira-bridge) (posts review findings to Jira)

[Code](https://modelshifts.com/work/) · 12 November 2026 · v1 · Python · [MIT](https://opensource.org/license/mit) · [CI](https://github.com/ModelShifts/pr-reviewer/actions/workflows/ci.yml)

## The problem

Imagine a pull request with a one-line change that subtracts the same reserved quantity twice. A reviewer who is already late opens the diff, looks for that kind of mistake, and writes a comment on the line.

The useful result is not a model that “reviews code.” It is a comment on the exact line, naming the bug and a fix, that a person can accept or dismiss.

## What we built

The pipeline is five steps. A pull request (or a unified diff on disk) is reviewed in short hunks, against a YAML rules file that sets what to check, what to ignore, and how strict to be. Inline comments land on the lines they refer to; comments on lines that are not in the diff are dropped. One summary comment sits at the top. The backend can be a hosted API or an open model on your own machine. An optional companion, jira-bridge, posts those findings to Jira when it is enabled.

![Pipeline: a pull request diff is fetched; a YAML rules file sets what to check; a code model reviews each hunk; inline comments land on the lines they refer to, plus one summary. Optional companion: jira-bridge.](https://modelshifts.com/media/work/pr-reviewer/pipeline.svg)

The same five steps, whether the backend is your API or open models on your own machine. Companion: jira-bridge, which posts review findings to Jira when it is enabled.

![Left: CLI output of pr-reviewer review, with inline comments and a summary. Right: the same comments in the Gradio demo. Model: Qwen2.5-Coder-7B-Instruct Q4_K_M. No GitHub identity is shown.](https://modelshifts.com/media/work/pr-reviewer/before-after.webp)

CLI review and the local Gradio demo. Comments produced by Qwen2.5-Coder-7B-Instruct (Q4_K_M).

A general “this looks risky” comment at the bottom of a pull request is easy to scroll past. A comment on the exact line, naming the bug and a fix, is something a person can accept, edit, or dismiss — and the rules file is how a team says what is worth commenting on.

## Results

On six diffs — five with a seeded bug, and one documentation-only change that the default rules ignore — the API backend with Qwen2.5-Coder-7B-Instruct (Q4_K_M) found 5 of 5 seeded bugs. The documentation-only diff produced no comments, as intended.

A finding counts as found when a comment is on the same file and side, the line is within the finding’s window, and the comment body contains one of the expected keywords (and does not agree with the bug instead of reporting it).

| Diff | Bug | Path | Expected findings |
| --- | --- | --- | --- |
| picker-index.diff | off-by-one (index <= len(bins)) | picker.py | 1 |
| double-subtract.diff | reserved quantity subtracted twice | stock.py | 1 |
| query-concat.diff | SQL built by string formatting | query.py | 1 |
| total-returns-count.diff | len(items) where a total was wanted | totals.py | 1 |
| discount-formula.diff | price / percent where a fraction was wanted | billing.py | 1 |
| docs-only.diff | documentation only (ignored) | — | 0 |
| **All** | **5 of 5 seeded bugs found** |  | **5 + 0** |

**Methodology.** Six diffs in samples/diffs, with ground truth in the matching *.truth.json files. API backend, Qwen2.5-Coder-7B-Instruct (Q4_K_M) behind a local llama.cpp server. The raw table is in [docs/eval.md](https://github.com/ModelShifts/pr-reviewer/blob/main/docs/eval.md). The script is [scripts/evaluate.py](https://github.com/ModelShifts/pr-reviewer/blob/main/scripts/evaluate.py); the samples are in [samples/](https://github.com/ModelShifts/pr-reviewer/tree/main/samples).



## Run it yourself

```bash
git clone https://github.com/ModelShifts/pr-reviewer
cd pr-reviewer && python -m venv .venv && source .venv/bin/activate
pip install -e ".[api,demo]" && pr-reviewer serve
```

The 7B figures used a Q4_K_M GGUF through llama.cpp (API backend). On the Hugging Face backend, a CPU-only machine should set MS_HF_MODEL=Qwen/Qwen2.5-Coder-1.5B-Instruct. Python 3.11 or newer.

[View the code on GitHub](https://github.com/ModelShifts/pr-reviewer)

[Work](https://modelshifts.com/work/) · [Home](https://modelshifts.com/)