mroberts/claude-plugin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move the code and terraform audits into the reviews plugin

Browse Source 
Copy the standalone code-review and terraform-review skills into
plugins/reviews as audit-code and audit-terraform. The rename separates the
automated, linter-driven audits from the guided review-pr walkthrough that
already lived here.

Resolve bundled script paths through ${SKILL_DIR}, exported in a new step 0.
CLAUDE_PLUGIN_ROOT is not set in the Bash tool environment, so the obvious
substitution would have expanded to nothing and broken every collection
script invocation.

Replace the PLAN and DESIGN docs with READMEs written from the current
SKILL.md and scripts. The old docs had drifted badly: they named semgrep
where the code calls opengrep, scoped five review agents where there are
now eight, and predated Lua, PowerShell, and GitHub Actions support.

Add CONSISTENCY_NORMS to the audit-terraform agent inputs. The collection
script writes consistency_norms.json and the agent prompt declares it, but
SKILL.md never listed it, leaving the variable unsubstituted.

Drop the --ingest-verdicts instruction from both skills. review_stats.py
parses no arguments, so the ref-mode verdict template it told users to feed
back could never be read.

Point audit-terraform's smoke test at README.md and resolve its fixture
paths relative to the test file rather than an absolute home directory.

Tests: 197 passing (audit-code), 106 passing (audit-terraform).
This commit is contained in:
mroberts
2026-07-21 11:11:05 -05:00
parent 600c1fef86
commit f5934181ec
179 changed files with 20779 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/reviews/skills/audit-terraform/README.md
+292
View File
@@ -0,0 +1,292 @@
# audit-terraform
Automated, tool-driven audit of terraform / terragrunt changes.
A mechanical Python collection script builds a manifest of the change (plan
output, diff-touched resources, module graph, scanner findings), slices that
manifest per reviewer, and the skill fans out four parallel LLM subagents
against those slices. Output is a walkthrough of what the change does plus
findings grouped by severity.
Ships in the `reviews` plugin of the `mroberts` marketplace, alongside
`review-pr`. The two are different tools: `review-pr` builds a guided
briefing so a *human* can read a PR; `audit-terraform` runs the scanners and
the review agents itself.
## When it runs
Auto-activates on phrasing like "audit this terraform", "check this
terragrunt change", or the explicit `/audit-terraform` invocation.
### Modes
| Invocation | Mode | Diff target | Output |
|--------------------------------|-------|----------------------------|---------------------------------|
| `/audit-terraform` | local | working tree vs base | interactive walkthrough in chat |
| `/audit-terraform <ref>` | ref | `<ref>` vs base | `audit-terraform-<short>.md` |
| `/audit-terraform <pr_number>` | ref | PR head ref vs base | same as above |
Local mode runs against the user's current repo. Ref/PR mode isolates the
checkout first — a git worktree under `~/.claude/cache/audit-terraform/<short-ref>/`
when the cwd is a git checkout of the repo, otherwise a fresh
`gh repo clone` into the same path. The clone path is what makes the skill
usable from jj workspaces, where `gh`/`git` autodetection fails.
The base ref is resolved automatically: `origin/HEAD` if the symbolic ref
exists, else `origin/main` or `origin/master`, else the literal branch name.
`origin` is fetched first so the comparison is against the remote tip rather
than a stale local branch.
## Pipeline
1. **Collection** — `scripts/collect-changes.py --repo --base --head
--output-dir --mode`. Everything below happens inside this one script.
2. **Diff scan** — `git diff --unified=0 <base>...<head>` for changed
`.tf` / `.tf.json` / `.hcl` files, then per-file changed line ranges are
mapped through `hcl_diff.py` (python-hcl2) to the resource/module blocks
they touch.
3. **Plan units** — `module_graph.py` builds the module→callsite graph;
`resolve_plan_units.py` maps each changed directory to the directories
that can actually be planned (a changed shared module resolves to its
callsites). Modules with no callsites are recorded as an error and
reviewed diff-only.
4. **Plan execution** — up to 8 plan units run concurrently. `plan_runner.py`
picks `terragrunt` if the unit has a `terragrunt.hcl`, otherwise `tofu`,
then runs `<tool> init -input=false -no-color` and `<tool> plan
-input=false -no-color`. Full stdout lands in `plans/<dir>.txt`;
`plan_output.py` parses it into resource addresses/actions and the
`Plan: N to add, N to change, N to destroy` summary.
5. **First-pass scanners** —
- `trivy config --quiet --format json <changed dirs>`, normalized into
`trivy_findings` (check id, title, severity, message, file, line range,
resource type). A trivy failure is a recorded error, not a hard stop.
- `tflint --format json --chdir <dir>` per changed terraform dir,
normalized into `tflint_findings` (rule, severity, message, file, line
range, doc link). Silently skipped if `tflint` is not on PATH.
6. **Catalog + context** — `catalog.py` merges plan hits and diff hits into
one entry per resource, tagged `source: plan | diff | both`.
`source_lookup.py` attaches the block header, an evidence line, key
attributes, and review context so agents rarely have to open source files.
7. **Reference sets** — `reference_set.py` computes peer directories for each
changed dir (region peers and same-component cross-env for the
`live/<env>/<region>/<component>` layout) plus precomputed consistency
norms.
8. **Manifest slicing** — `slicing.py` writes a per-agent subset so each
subagent only sees what it needs. AWS-lane slices filter the catalog to
`aws_*` types; the tf-hygiene slice omits trivy findings; the walkthrough
slice omits scanner findings entirely.
9. **Fan-out** — four Task subagents dispatched in a single message, each
given its manifest slice, `REPO`, and a `findings-<agent>.json` output
path.
10. **Aggregation** — the walkthrough payload is loaded separately; remaining
findings are deduplicated on `{resource, control}` (first wins, loser
recorded in `also_flagged_by`) and grouped by severity.
11. **Telemetry** — one `scripts/log-run.py` call appends a row per agent.
12. **Output** — chat message in local mode, markdown report in ref mode.
### Exit behaviour
`collect-changes.py` returns non-zero if the git diff fails or any plan unit
failed to plan. On non-zero, the skill reports `manifest.json` `errors[]`
verbatim and does **not** run subagents. Zero changed HCL files is exit 0
with a `no terraform/hcl files changed` error entry.
## External tools
Shelled out to by the scripts:
| Tool | Used by | Required |
|---|---|---|
| `git` | diff scanning, base-ref resolution | yes |
| `tofu` | init/plan for non-terragrunt units | yes, if any plan unit is plain terraform |
| `terragrunt` | init/plan for units with a `terragrunt.hcl` | yes, if any plan unit is terragrunt |
| `trivy` | `trivy config` first-pass scan | yes — a failure is recorded as an error |
| `tflint` | per-dir lint | optional — skipped if absent |
`gh` is used by the skill procedure (not the scripts) to resolve the repo
name-with-owner, look up PR head refs, and clone in ref/PR mode.
Note the plan runner invokes `tofu` specifically. There is no `terraform`
binary fallback — install OpenTofu, or symlink `tofu` to `terraform`.
### Install
Homebrew covers all of them:
```
brew install opentofu terragrunt trivy tflint gh
```
Upstream instructions: <https://opentofu.org/docs/intro/install/>,
<https://terragrunt.gruntwork.io/docs/getting-started/install/>,
<https://trivy.dev/latest/getting-started/installation/>,
<https://github.com/terraform-linters/tflint>, <https://cli.github.com/>.
Python dependencies are in `requirements.txt`: `python-hcl2`,
`beautifulsoup4`, `requests`, `pytest`.
## Subagents
Prompts live in `agents/`. Four are the default set.
| Agent | Model | Scope |
|---|---|---|
| `walkthrough-reviewer` | Sonnet | Reviewer-facing summary: 3–6 sentence overview plus a per-plan-unit summary classified `substantive` or `trivial`. Describes, does not grade. Emits **no** findings — its output JSON is `{overview, plan_units[]}`. |
| `aws-bp-reviewer` | Sonnet | Primary security lane. Reads `trivy_findings` first, triages and suppresses duplicates/low-signal hits, then adds only contextual AWS best-practice findings trivy is likely to miss (encryption tradeoffs, retention/lifecycle, backup posture, multi-AZ, IAM least-privilege nuance, logging blind spots, deletion protection). |
| `consistency-reviewer` | Sonnet | Repo-internal only. Compares each changed dir against its reference set and precomputed norms — missing patterns peers all use, naming/variable drift, missing standard tags, missing `kms_key_arn`. Evidence must cite at least two peer dirs plus the diverging file:line. |
| `tf-hygiene-reviewer` | Haiku 4.5 | Module hygiene and maintainability, strictly non-security. Consumes `tflint_findings`, then adds what tflint can't infer: variable/output `type` and `description`, `sensitive` flags, version pinning, lifecycle blocks, terragrunt patterns. Runs on Haiku because tflint already did the mechanical work. |
### Legacy reviewers
`agents/fsbp-reviewer.md` and `agents/cis-reviewer.md` are benchmark-specific
prompts kept for explicit fallback or cross-check work. They are **not** part
of the default path — `aws-bp-reviewer` replaced them once trivy became the
first-pass detector. Use them only when asked to confirm findings against
FSBP or CIS specifically.
`collect-changes.py` still writes `manifest-fsbp.json` and
`manifest-cis.json` slices so those prompts can be run without re-collecting.
### Findings contract
Every finding-emitting agent writes:
```json
{
"agent": "aws-bp-reviewer",
"started_at": "...",
"finished_at": "...",
"skipped_resources": [{"address": "...", "reason": "not AWS"}],
"findings": [
{
"resource": "aws_s3_bucket.audit_logs",
"dir": "live/prod/us-east-1/audit",
"control": "TRIVY AVD-AWS-0089",
"severity": "critical | high | medium | low",
"issue": "...",
"evidence": "main.tf:21 — ...",
"fix": "..."
}
]
}
```
## Controls data
`data/controls/` holds JSON snapshots of AWS security control catalogues,
indexed by terraform resource type:
- `fsbp.json` — AWS Foundational Security Best Practices (368 controls in the
committed snapshot)
- `cis.json` — CIS AWS Foundations Benchmark (65 controls)
- `meta.json` — source URL and fetch timestamp per file
Refresh from the AWS Security Hub docs:
```
python ${SKILL_DIR}/scripts/refresh-controls.py [--output-dir DIR]
```
`refresh-controls.py` fetches the FSBP standard index and the CIS benchmark
page over HTTPS, parses them with `scrape_fsbp.py` / `scrape_cis.py`
(BeautifulSoup), validates rows through `controls_schema.py`, and rewrites all
three files. It prints the control counts it wrote. `--output-dir` defaults to
`data/controls/` inside the skill.
The cache is treated as stale past 30 days; the benchmark agents check
`meta.json` and fall back to a live fetch if a file is missing, empty, or
stale. Only the legacy `fsbp-reviewer` / `cis-reviewer` prompts consume this
data — the default four-agent path does not read it, since trivy supplies
first-pass benchmark detection.
## Output artifacts
The output directory is `<REPO>/.audit-terraform/` in ref mode and
`~/.claude/cache/audit-terraform/local-<UTC-timestamp>/` in local mode.
| File | Written by | Contents |
|---|---|---|
| `manifest.json` | collection | Full manifest — base/head refs, mode, default branch, changed source dirs, plan units, resource catalog, trivy + tflint findings, module graph, errors |
| `manifest-<agent>.json` | collection | Per-agent slice for `fsbp`, `cis`, `aws-bp`, `consistency`, `tf-hygiene`, `walkthrough` |
| `trivy-findings.json` | collection | Raw `trivy config` JSON |
| `tflint-findings.json` | collection | Raw tflint JSON, keyed `{"by_dir": {...}}` |
| `reference_sets.json` | collection | Peer dirs per changed dir (consistency agent input) |
| `consistency_norms.json` | collection | Precomputed norms derived from the reference sets |
| `plans/<dir>.txt` | collection | Full plan stdout per plan unit (`.` becomes `root`, `/` becomes `_`) |
| `findings-<agent>.json` | subagents | Findings, or the walkthrough payload for `walkthrough-reviewer` |
Ref mode additionally writes `<REPO>/audit-terraform-<short-ref>.md`: summary
table, walkthrough (substantive then trivial plan units), per-dir plan
summaries, findings by severity → category → resource, consistency findings,
and skipped resources. The worktree/clone path is left in place and reported
so follow-up review can use it.
## Telemetry
Append-only JSONL at `~/.claude/cache/audit-terraform/runs.jsonl`. Two record
kinds, both defined in `scripts/telemetry.py`:
- `subagent_run` — `run_id`, `repo`, `mode`, `agent`, `model`,
`input_tokens`, `output_tokens`, `duration_ms`, `finding_count`
- `verdict` — `run_id`, `agent`, `rule_id`, `file`, `line`, `verdict`
(`kept` | `dismissed` | `false_positive`), `notes`
Runs are logged with one call after the fan-out completes. The orchestrator
supplies token/duration metadata per agent; `log-run.py` reads the finding
count off disk from each `findings-<agent>.json`:
```
echo '{"aws-bp-reviewer": {"model":"sonnet","input_tokens":N,"output_tokens":N,"duration_ms":N}}' \
| python ${SKILL_DIR}/scripts/log-run.py \
--output-dir <OUTPUT> --run-id <hex8> --repo <REPO> \
--mode <local|ref> --usage-json -
```
`--log-path` overrides the default log location.
Read the log back with:
```
python ${SKILL_DIR}/scripts/review_stats.py
```
It prints JSON with `by_agent` (tokens, duration, runs, verdict counts,
`precision` = kept/total, `tokens_per_kept`), `by_rule` keyed
`<agent>/<rule_id>`, and a distinct `runs` count. It takes no arguments and
always reads the default log path.
Verdicts are how precision gets measured. Local mode prompts for them
interactively and appends via `telemetry.append_verdict`. Ref mode does not
collect verdicts, so precision figures reflect local runs alone.
## Development
Tests live in `tests/` — 106 of them, covering the diff and HCL parsers, plan
unit resolution, plan output parsing, manifest and slicing shapes, scanner
normalization, the controls scrapers and schema, telemetry, stats, and agent
prompt invariants. One test in `tests/test_cli.py` is marked `integration`
and needs real `tofu` / `terragrunt`.
Run them with:
```
cd ${SKILL_DIR} && python -m pytest
```
`uv run pytest` does **not** work here: `pyproject.toml` carries only
`[tool.ruff.lint.per-file-ignores]` and `[tool.pytest.ini_options]`, with no
`[project]` table, so uv exits with ``No `project` table found``. Test imports
resolve through `tests/conftest.py`, which puts the skill root on `sys.path`.
Lint with ruff; `collect-changes.py` and `refresh-controls.py` are exempted
from `E402` because both mutate `sys.path` before importing sibling modules.
### Layout
```
SKILL.md procedure — source of truth for behaviour
agents/ subagent system prompts (4 default + 2 legacy)
data/controls/ FSBP/CIS snapshots + fetch metadata
scripts/ collection pipeline, scrapers, telemetry
tests/ pytest suite + HTML fixtures for the scrapers
```