docs: tier-1 language additions plan
This commit is contained in:
@@ -0,0 +1,380 @@
|
|||||||
|
# Tier-1 Language Additions Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax.
|
||||||
|
|
||||||
|
**Goal:** Add 11 languages (19 extensions) to shush — Terraform/HCL, Solidity, Zig, Elixir, Nix, Protobuf, GraphQL, OCaml, Haskell, Clojure — all already covered by chroma lexers (verified lossless + comment-stripping).
|
||||||
|
|
||||||
|
**Architecture:** Add `languageMap` + `GetLanguageName` entries (gates support + feeds the legacy fallback), extend `resolveLexer` overrides for extensions chroma's `Match` misses, and add per-language golden fixtures. No engine logic changes.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.25, chroma v2.27, existing golden harness.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- **No code comments anywhere** in Go files (hard project rule). testdata fixtures are comment-bearing input — that is the point.
|
||||||
|
- **jj only, never git.** Commit `jj describe -m "..."` then `jj new`.
|
||||||
|
- Gate: `make check` clean before each commit.
|
||||||
|
- Module path `github.com/carlosarraes/shush`.
|
||||||
|
- Goldens are hand-specified from this plan; run tests WITHOUT any update flag. If a language's fixture fails (its lexer tokenises differently), STOP and report the language + got/want; do not fudge the golden.
|
||||||
|
|
||||||
|
## Extension → lexer map (reference)
|
||||||
|
|
||||||
|
| ext | chroma resolution |
|
||||||
|
|-----|-------------------|
|
||||||
|
| tf | Match → Terraform |
|
||||||
|
| hcl | Match → HCL |
|
||||||
|
| tfvars | override → `lexers.Get("Terraform")` |
|
||||||
|
| sol | Match → Solidity |
|
||||||
|
| zig | Match → Zig |
|
||||||
|
| ex, exs | Match → Elixir |
|
||||||
|
| nix | Match → Nix |
|
||||||
|
| proto | Match → Protocol Buffer |
|
||||||
|
| graphql, graphqls | Match → GraphQL |
|
||||||
|
| ml, mli | Match → OCaml |
|
||||||
|
| hs | Match → Haskell |
|
||||||
|
| clj, edn | Match → Clojure |
|
||||||
|
| cljs, cljc | override → `lexers.Get("Clojure")` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
- `internal/processor/languages.go` — MODIFY: add entries to `languageMap` and `GetLanguageName`.
|
||||||
|
- `internal/processor/engine.go` — MODIFY: extend `resolveLexer` overrides.
|
||||||
|
- `internal/processor/engine_test.go` — MODIFY: extend `TestResolveLexer`.
|
||||||
|
- `internal/processor/tier1_test.go` — CREATE: behavior goldens (reuses `runMultilineCase` from `multiline_test.go`).
|
||||||
|
- `internal/processor/testdata/multiline/tier1_*` — CREATE: fixture + `.golden` pairs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Register languages + lexer routing
|
||||||
|
|
||||||
|
**Files:** `languages.go`, `engine.go`, `engine_test.go`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Extend TestResolveLexer (failing)**
|
||||||
|
|
||||||
|
In `internal/processor/engine_test.go`, add these rows to the `TestResolveLexer` `cases` slice (after the existing rows):
|
||||||
|
|
||||||
|
```go
|
||||||
|
{"a.tf", "Terraform"},
|
||||||
|
{"a.hcl", "HCL"},
|
||||||
|
{"a.tfvars", "Terraform"},
|
||||||
|
{"a.sol", "Solidity"},
|
||||||
|
{"a.zig", "Zig"},
|
||||||
|
{"a.ex", "Elixir"},
|
||||||
|
{"a.exs", "Elixir"},
|
||||||
|
{"a.nix", "Nix"},
|
||||||
|
{"a.proto", "Protocol Buffer"},
|
||||||
|
{"a.graphql", "GraphQL"},
|
||||||
|
{"a.graphqls", "GraphQL"},
|
||||||
|
{"a.ml", "OCaml"},
|
||||||
|
{"a.mli", "OCaml"},
|
||||||
|
{"a.hs", "Haskell"},
|
||||||
|
{"a.clj", "Clojure"},
|
||||||
|
{"a.edn", "Clojure"},
|
||||||
|
{"a.cljs", "Clojure"},
|
||||||
|
{"a.cljc", "Clojure"},
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run to verify it fails**
|
||||||
|
|
||||||
|
Run: `cd /home/mroberts/tmp/review/shush-fork && go test ./internal/processor -run TestResolveLexer`
|
||||||
|
Expected: FAIL — `a.tfvars`, `a.cljs`, `a.cljc` resolve to empty (Match misses them).
|
||||||
|
|
||||||
|
- [ ] **Step 3: Extend resolveLexer overrides**
|
||||||
|
|
||||||
|
In `internal/processor/engine.go`, replace the `if ext == "less" || ext == "sass" {` block with a switch:
|
||||||
|
|
||||||
|
```go
|
||||||
|
var l chroma.Lexer
|
||||||
|
switch ext {
|
||||||
|
case "less", "sass":
|
||||||
|
l = lexers.Get("SCSS")
|
||||||
|
case "tfvars":
|
||||||
|
l = lexers.Get("Terraform")
|
||||||
|
case "cljs", "cljc":
|
||||||
|
l = lexers.Get("Clojure")
|
||||||
|
default:
|
||||||
|
l = lexers.Match("f." + ext)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
(Keep the surrounding `ext := ...`, the `if l == nil { return nil }`, and `return chroma.Coalesce(l)` unchanged.)
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add languageMap entries**
|
||||||
|
|
||||||
|
In `internal/processor/languages.go`, add these entries to the `languageMap` (place near related groups; exact placement does not matter):
|
||||||
|
|
||||||
|
```go
|
||||||
|
"tf": {LineComment: "#", AlternateLineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||||
|
"hcl": {LineComment: "#", AlternateLineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||||
|
"tfvars": {LineComment: "#", AlternateLineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||||
|
"sol": {LineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||||
|
"zig": {LineComment: "//"},
|
||||||
|
"ex": {LineComment: "#"},
|
||||||
|
"exs": {LineComment: "#"},
|
||||||
|
"nix": {LineComment: "#", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||||
|
"proto": {LineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||||
|
"graphql": {LineComment: "#"},
|
||||||
|
"graphqls": {LineComment: "#"},
|
||||||
|
"ml": {BlockComment: &types.BlockComment{Start: "(*", End: "*)"}},
|
||||||
|
"mli": {BlockComment: &types.BlockComment{Start: "(*", End: "*)"}},
|
||||||
|
"hs": {LineComment: "--", BlockComment: &types.BlockComment{Start: "{-", End: "-}"}},
|
||||||
|
"clj": {LineComment: ";"},
|
||||||
|
"cljs": {LineComment: ";"},
|
||||||
|
"cljc": {LineComment: ";"},
|
||||||
|
"edn": {LineComment: ";"},
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add GetLanguageName entries**
|
||||||
|
|
||||||
|
In `GetLanguageName`'s `names` map add:
|
||||||
|
|
||||||
|
```go
|
||||||
|
"tf": "Terraform",
|
||||||
|
"hcl": "HCL",
|
||||||
|
"tfvars": "Terraform",
|
||||||
|
"sol": "Solidity",
|
||||||
|
"zig": "Zig",
|
||||||
|
"ex": "Elixir",
|
||||||
|
"exs": "Elixir",
|
||||||
|
"nix": "Nix",
|
||||||
|
"proto": "Protocol Buffer",
|
||||||
|
"graphql": "GraphQL",
|
||||||
|
"graphqls": "GraphQL",
|
||||||
|
"ml": "OCaml",
|
||||||
|
"mli": "OCaml",
|
||||||
|
"hs": "Haskell",
|
||||||
|
"clj": "Clojure",
|
||||||
|
"cljs": "ClojureScript",
|
||||||
|
"cljc": "Clojure",
|
||||||
|
"edn": "EDN",
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Run resolver test + build**
|
||||||
|
|
||||||
|
Run: `go test ./internal/processor -run TestResolveLexer && go build ./...`
|
||||||
|
Expected: PASS; build clean.
|
||||||
|
|
||||||
|
- [ ] **Step 7: `make check` then commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
jj describe -m "feat(languages): register tier-1 languages and chroma routing"
|
||||||
|
jj new
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Behavior goldens
|
||||||
|
|
||||||
|
**Files:** `internal/processor/tier1_test.go` (create), `testdata/multiline/tier1_*` (create)
|
||||||
|
|
||||||
|
**Interfaces consumed:** `runMultilineCase`, `multilineCase` (from `multiline_test.go`, same package).
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the case table (failing)**
|
||||||
|
|
||||||
|
Create `internal/processor/tier1_test.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package processor
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestTier1Languages(t *testing.T) {
|
||||||
|
cases := []multilineCase{
|
||||||
|
{name: "terraform", file: "tier1_terraform.tf"},
|
||||||
|
{name: "solidity", file: "tier1_solidity.sol"},
|
||||||
|
{name: "zig", file: "tier1_zig.zig"},
|
||||||
|
{name: "elixir", file: "tier1_elixir.ex"},
|
||||||
|
{name: "nix", file: "tier1_nix.nix"},
|
||||||
|
{name: "protobuf", file: "tier1_protobuf.proto"},
|
||||||
|
{name: "graphql", file: "tier1_graphql.graphql"},
|
||||||
|
{name: "ocaml", file: "tier1_ocaml.ml"},
|
||||||
|
{name: "haskell", file: "tier1_haskell.hs"},
|
||||||
|
{name: "clojure", file: "tier1_clojure.clj"},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
runMultilineCase(t, tc)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test ./internal/processor -run TestTier1Languages`
|
||||||
|
Expected: FAIL — missing `testdata/multiline/tier1_terraform.tf`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Create fixtures + goldens with exact bytes**
|
||||||
|
|
||||||
|
`tier1_terraform.tf`:
|
||||||
|
```
|
||||||
|
resource "a" "b" { # h
|
||||||
|
x = 1 // s
|
||||||
|
/* block
|
||||||
|
cmt */
|
||||||
|
y = 2
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`tier1_terraform.tf.golden`:
|
||||||
|
```
|
||||||
|
resource "a" "b" {
|
||||||
|
x = 1
|
||||||
|
y = 2
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_solidity.sol`:
|
||||||
|
```
|
||||||
|
contract C {
|
||||||
|
// line
|
||||||
|
/* block
|
||||||
|
cmt */
|
||||||
|
uint x;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`tier1_solidity.sol.golden`:
|
||||||
|
```
|
||||||
|
contract C {
|
||||||
|
uint x;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_zig.zig`:
|
||||||
|
```
|
||||||
|
const x = 1; // line
|
||||||
|
// c2
|
||||||
|
pub fn main() void {}
|
||||||
|
```
|
||||||
|
`tier1_zig.zig.golden`:
|
||||||
|
```
|
||||||
|
const x = 1;
|
||||||
|
pub fn main() void {}
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_elixir.ex`:
|
||||||
|
```
|
||||||
|
defmodule M do
|
||||||
|
# line
|
||||||
|
def f, do: 1
|
||||||
|
end
|
||||||
|
```
|
||||||
|
`tier1_elixir.ex.golden`:
|
||||||
|
```
|
||||||
|
defmodule M do
|
||||||
|
def f, do: 1
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_nix.nix`:
|
||||||
|
```
|
||||||
|
{
|
||||||
|
# line
|
||||||
|
x = 1;
|
||||||
|
/* block
|
||||||
|
cmt */
|
||||||
|
y = 2;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`tier1_nix.nix.golden`:
|
||||||
|
```
|
||||||
|
{
|
||||||
|
x = 1;
|
||||||
|
y = 2;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_protobuf.proto`:
|
||||||
|
```
|
||||||
|
syntax = "proto3";
|
||||||
|
// line
|
||||||
|
message M {
|
||||||
|
/* block
|
||||||
|
cmt */
|
||||||
|
int32 x = 1;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`tier1_protobuf.proto.golden`:
|
||||||
|
```
|
||||||
|
syntax = "proto3";
|
||||||
|
message M {
|
||||||
|
int32 x = 1;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_graphql.graphql`:
|
||||||
|
```
|
||||||
|
type Q {
|
||||||
|
# line
|
||||||
|
f: Int
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`tier1_graphql.graphql.golden`:
|
||||||
|
```
|
||||||
|
type Q {
|
||||||
|
f: Int
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_ocaml.ml`:
|
||||||
|
```
|
||||||
|
let x = 1
|
||||||
|
(* block
|
||||||
|
cmt *)
|
||||||
|
let y = 2
|
||||||
|
```
|
||||||
|
`tier1_ocaml.ml.golden`:
|
||||||
|
```
|
||||||
|
let x = 1
|
||||||
|
let y = 2
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_haskell.hs`:
|
||||||
|
```
|
||||||
|
main = putStrLn "x" -- line
|
||||||
|
{- block
|
||||||
|
cmt -}
|
||||||
|
f = 1
|
||||||
|
```
|
||||||
|
`tier1_haskell.hs.golden`:
|
||||||
|
```
|
||||||
|
main = putStrLn "x"
|
||||||
|
f = 1
|
||||||
|
```
|
||||||
|
|
||||||
|
`tier1_clojure.clj`:
|
||||||
|
```
|
||||||
|
(ns app) ; line
|
||||||
|
(def x 1)
|
||||||
|
```
|
||||||
|
`tier1_clojure.clj.golden`:
|
||||||
|
```
|
||||||
|
(ns app)
|
||||||
|
(def x 1)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test ./internal/processor -run TestTier1Languages -v`
|
||||||
|
Expected: PASS — 10 subtests. If a language fails, STOP and report got/want (a lexer may include/exclude boundary whitespace differently); do not edit the golden to force green.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Full suite + static build**
|
||||||
|
|
||||||
|
Run: `go test ./... && CGO_ENABLED=0 go build -o /tmp/shush_t1 ./cmd/shush && file /tmp/shush_t1`
|
||||||
|
Expected: all green; statically linked.
|
||||||
|
|
||||||
|
- [ ] **Step 6: `make check` then commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
jj describe -m "test(languages): tier-1 language comment-removal goldens"
|
||||||
|
jj new
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review (completed during authoring)
|
||||||
|
|
||||||
|
- **Coverage:** 19 extensions registered (Task 1), 11 languages behavior-tested (Task 2). Every extension asserted in `TestResolveLexer`; every language in `TestTier1Languages`.
|
||||||
|
- **Placeholder scan:** exact map entries, exact resolveLexer switch, exact fixture bytes. No TBD.
|
||||||
|
- **Type consistency:** `resolveLexer` switch matches existing signature; `languageMap`/`names` use existing `types.Language`/`types.BlockComment`; `runMultilineCase`/`multilineCase` reused unchanged.
|
||||||
|
- **Comment styles** verified via chroma spike (all 11 languages: lossless + comments removed). `.v` deliberately excluded (ambiguous Coq/V/Verilog).
|
||||||
Reference in New Issue
Block a user