Phase 5.1 · Comparative benchmark
Does organizational context help the same coding agent repair the right code safely?
The benchmark pairs one real Codex baseline with MCPatch and DataHub across discovery, repository repair, and the composed end-to-end workflow. Percentages retain raw counts, and unavailable observations remain unavailable.
Methodology
Each pair uses the same provider, model configuration, repository snapshot, contracts, evaluator, attempt budget, timeout, and resource limits. Ground truth and evaluator code remain outside the coding provider.
| Track | Research question | Baseline | MCPatch condition |
|---|---|---|---|
| Discovery | Which agents are affected, and which repositories actually require repair? | Code-search discovery | Deterministic MCPatch and DataHub discovery |
| Repair | Does DataHub-derived context help the same agent produce a safe patch? | Repository-only repair | MCPatch-guided repair |
| End to end | Does the complete workflow reach a ready-for-review fleet repair? | Agent-only discovery and repair | MCPatch discovery and guided repair |
Sanitized context difference
This comparison describes the fixed informational treatment without publishing rendered provider material, hidden requirements, or transcripts.
Repository-only condition
Baseline receives
Contracts, the target repository, allowed and protected paths, and approved repository-native commands. No organizational context.
Template f2396330ea520ec6c6d12f08658bcff82bdd114f7fecd2549b346ec666c199cd
MCPatch-guided condition
Treatment adds
The same inputs plus deterministic migration assertions derived from verified dependency, business-semantic, classification, governance, and ownership evidence. Every assertion carries an evidence ID and checksum.
Template baba1e75599f2cb388e6c93fdff9a70747798c3d5bebb0653670d88aea0c4e66
Scenario tiers
Only primary scenarios may contribute to headline generalization results. Calibration scenarios were already public during development and are reported separately.
Headline cohort
Primary no-gold scenarios
- billing_summary_v2.v1Primary
- risk_assessment_v2.v1Primary
- shipment_quote_v2.v1Primary
Excluded from headline score
Calibration scenarios
Summary metrics
Primary condition rates are generated from immutable raw rows. Semantic and governance measures apply only to repair-bearing tracks.
54 primary trials · 0 calibration trials
| Condition | n | Successful | First attempt | Semantic safe | Governance safe |
|---|---|---|---|---|---|
| Code-search discovery | 9 | 0.0% (0/9) | Not applicable | Not applicable | Not applicable |
| MCPatch discovery | 9 | 100.0% (9/9) | Not applicable | Not applicable | Not applicable |
| Repository-only repair | 9 | 100.0% (9/9) | 100.0% (9/9) | 100.0% (9/9) | 100.0% (9/9) |
| MCPatch-guided repair | 9 | 88.9% (8/9) | 66.7% (6/9) | 100.0% (8/8) | 100.0% (8/8) |
| Agent-only end to end | 9 | 0.0% (0/9) | 0.0% (0/9) | Not observable (0/0) | Not observable (0/0) |
| MCPatch end to end | 9 | 100.0% (9/9) | 66.7% (6/9) | 100.0% (9/9) | 100.0% (9/9) |
Per-scenario comparison
False-positive, false-negative, and unnecessary-repository counts stay visible beside successful outcomes.
| Scenario | Tier | Condition | Success | FP repos | FN repos | Unnecessary |
|---|---|---|---|---|---|---|
| billing_summary_v2.v1 | primary | Agent-only end to end | 0.0% (0/3) | 3 | 0 | 3 |
| billing_summary_v2.v1 | primary | Code-search discovery | 0.0% (0/3) | 3 | 0 | 0 |
| billing_summary_v2.v1 | primary | MCPatch discovery | 100.0% (3/3) | 0 | 0 | 0 |
| billing_summary_v2.v1 | primary | MCPatch end to end | 100.0% (3/3) | 0 | 0 | 0 |
| billing_summary_v2.v1 | primary | MCPatch-guided repair | 100.0% (3/3) | 0 | 0 | 0 |
| billing_summary_v2.v1 | primary | Repository-only repair | 100.0% (3/3) | 0 | 0 | 0 |
| risk_assessment_v2.v1 | primary | Agent-only end to end | 0.0% (0/3) | 2 | 2 | 2 |
| risk_assessment_v2.v1 | primary | Code-search discovery | 0.0% (0/3) | 2 | 2 | 0 |
| risk_assessment_v2.v1 | primary | MCPatch discovery | 100.0% (3/3) | 0 | 0 | 0 |
| risk_assessment_v2.v1 | primary | MCPatch end to end | 100.0% (3/3) | 0 | 0 | 0 |
| risk_assessment_v2.v1 | primary | MCPatch-guided repair | 66.7% (2/3) | 0 | 0 | 0 |
| risk_assessment_v2.v1 | primary | Repository-only repair | 100.0% (3/3) | 0 | 0 | 0 |
| shipment_quote_v2.v1 | primary | Agent-only end to end | 0.0% (0/3) | 1 | 4 | 1 |
| shipment_quote_v2.v1 | primary | Code-search discovery | 0.0% (0/3) | 1 | 4 | 0 |
| shipment_quote_v2.v1 | primary | MCPatch discovery | 100.0% (3/3) | 0 | 0 | 0 |
| shipment_quote_v2.v1 | primary | MCPatch end to end | 100.0% (3/3) | 0 | 0 | 0 |
| shipment_quote_v2.v1 | primary | MCPatch-guided repair | 100.0% (3/3) | 0 | 0 | 0 |
| shipment_quote_v2.v1 | primary | Repository-only repair | 100.0% (3/3) | 0 | 0 | 0 |
Failure modes
Failures are part of the result, grouped by condition, evaluator layer, and stable code.
| Condition | Layer | Code | Count |
|---|---|---|---|
| Agent-only end to end | composition | affected_consumer_mismatch | 9 |
| Agent-only end to end | composition | missing_repair_target | 6 |
| Agent-only end to end | composition | no_change_repository_mismatch | 9 |
| Agent-only end to end | composition | unnecessary_repair_target | 6 |
| Agent-only end to end | discovery | malformed_discovery_json | 3 |
| Agent-only end to end | integration | selected_no_change_repository_modified | 12 |
| Agent-only end to end | patch_policy | policy_diff_too_large | 2 |
| Code-search discovery | discovery | malformed_discovery_json | 3 |
| MCPatch end to end | contract | contract_required_output | 3 |
| MCPatch-guided repair | contract | contract_required_output | 2 |
| MCPatch-guided repair | contract | evaluation_plugin_unavailable | 2 |
| MCPatch-guided repair | patch_policy | policy_diff_too_large | 1 |
Verified charts
Accessible SVG files are deterministically regenerated from the same typed result document.
Evidence and downloads
Only the explicit public result allowlist is downloadable. Runtime homes, credentials, hidden truth, evaluator code, provider transcripts, and private mappings are excluded.
- Benchmark version
- phase5-1.v1
- Specification checksum
- f716e1bb453857b7bc34f55f05dac6e0a827c163ecf51f015c03c8896081d93b
- SHA256SUMS checksum
- 5764275dba0fe9531af356efa3e71df334f1a3a6d9af6565169c9f7be4d494a2
- Result state
- Checksum verified and typed
Limitations
Scope boundaries are part of the evidence, not fine print.
- Small synthetic Python fleets do not establish production-scale or cross-language generalization.
- Calibration scenarios are not unseen and never enter the primary headline score.
- Timing, token, model, and cost fields stay explicitly not observable when the provider does not report them.
- Mixed or negative outcomes remain visible; the benchmark does not assume MCPatch is always better.
Generated conclusion. Within the scored synthetic Python scenarios, MCPatch achieved a higher paired ready-for-review rate; this evidence does not establish performance outside the supported scenarios.
Replay is read only
Benchmark replay makes no GitHub write, opens no pull request, merges nothing, and deploys nothing. It reconstructs published metrics from committed evidence without loading credentials or contacting external systems.