Architecture Lab
AI-assisted code review pipeline
Combining static analysis, security scanning and an LLM reviewer grounded in project guidelines into one set of review comments.
- LLMs
- Static analysis
- Security scanning
- RAG
Illustrative reference design built from general engineering practice. It is not the architecture of any employer's or client's production system.
Why it matters
LLM reviewers catch design and readability issues that linters miss, but they are slower, cost money and can be wrong. Deterministic tools are fast and precise but narrow. This design runs both and merges the results. It is the model behind CheckMate, my AI-assisted code review project.
Diagram
- Request / call
- Asynchronous
Diagram as text
Components
- Pull request event (input or output): Webhook
- Review orchestrator (control point): One run per commit
- Static analysis (service): Linters, type checks
- Security scan (service): Secrets, risky calls
- Project guidelines (data store): Indexed for retrieval
- Context builder (service): Diff + retrieved rules
- LLM reviewer (external system): Via a gateway
- Findings aggregator (control point): Dedupe, rank, filter
- Review comments (input or output): Inline + summary
Connections
- Pull request event → Review orchestrator
- Review orchestrator → Static analysis
- Review orchestrator → Security scan
- Review orchestrator → Context builder (asynchronous)
- Project guidelines → Context builder
- Context builder → LLM reviewer
- Static analysis → Findings aggregator
- Security scan → Findings aggregator
- LLM reviewer → Findings aggregator
- Findings aggregator → Review comments
How it flows
Receive the event
A pull request webhook is verified and keyed by repository and commit, so a redelivered event doesn't start a second review.
Run deterministic checks
Static analysis and a security scan run on the changed files and produce precise, line-level findings.
Build context
The diff is split per file and combined with the project's own guidelines, retrieved from an index, so the model reviews against house rules.
LLM review
The model returns structured findings (file, line, severity, explanation), not free text.
Aggregate
Findings are de-duplicated against the deterministic tools, ranked by severity, and low-confidence items are dropped.
Report
Inline comments plus one summary comment are posted back to the pull request.
Boundaries
Source code boundary
Code leaves the system only through the LLM gateway, after likely secrets are redacted; a self-hosted model can replace the provider for sensitive repositories.
Advisory, not blocking
AI findings are suggestions. Only deterministic checks are allowed to fail a build.
Tool isolation
Each analyser runs on its own, so one crashing tool doesn't stop the rest of the review.
Failure handling
- LLM times out or errors
- Post the deterministic findings anyway and note that the AI review was skipped.
- Very large diff
- Review per file within a size budget, and say which files were not reviewed.
- Duplicate webhook delivery
- The commit-level key makes the second run a no-op.
- Noisy or wrong AI comments
- Structured output, a confidence threshold and de-duplication keep only specific, located findings.
Trade-offs
Run deterministic tools and the LLM side by side
- Gain
- Precise findings arrive quickly while the LLM covers what the tools can't see.
- Cost
- Results from different sources must be reconciled.
Retrieve project guidelines
- Gain
- Reviews follow the team's own conventions rather than generic advice.
- Cost
- The guideline index has to be kept current.
Structured findings instead of prose
- Gain
- Comments can be placed on exact lines, de-duplicated and filtered.
- Cost
- Stricter prompts and output validation.