docs: tier-1 language additions plan
This commit is contained in:
@@ -0,0 +1,380 @@
|
||||
# Tier-1 Language Additions Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** Add 11 languages (19 extensions) to shush — Terraform/HCL, Solidity, Zig, Elixir, Nix, Protobuf, GraphQL, OCaml, Haskell, Clojure — all already covered by chroma lexers (verified lossless + comment-stripping).
|
||||
|
||||
**Architecture:** Add `languageMap` + `GetLanguageName` entries (gates support + feeds the legacy fallback), extend `resolveLexer` overrides for extensions chroma's `Match` misses, and add per-language golden fixtures. No engine logic changes.
|
||||
|
||||
**Tech Stack:** Go 1.25, chroma v2.27, existing golden harness.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **No code comments anywhere** in Go files (hard project rule). testdata fixtures are comment-bearing input — that is the point.
|
||||
- **jj only, never git.** Commit `jj describe -m "..."` then `jj new`.
|
||||
- Gate: `make check` clean before each commit.
|
||||
- Module path `github.com/carlosarraes/shush`.
|
||||
- Goldens are hand-specified from this plan; run tests WITHOUT any update flag. If a language's fixture fails (its lexer tokenises differently), STOP and report the language + got/want; do not fudge the golden.
|
||||
|
||||
## Extension → lexer map (reference)
|
||||
|
||||
| ext | chroma resolution |
|
||||
|-----|-------------------|
|
||||
| tf | Match → Terraform |
|
||||
| hcl | Match → HCL |
|
||||
| tfvars | override → `lexers.Get("Terraform")` |
|
||||
| sol | Match → Solidity |
|
||||
| zig | Match → Zig |
|
||||
| ex, exs | Match → Elixir |
|
||||
| nix | Match → Nix |
|
||||
| proto | Match → Protocol Buffer |
|
||||
| graphql, graphqls | Match → GraphQL |
|
||||
| ml, mli | Match → OCaml |
|
||||
| hs | Match → Haskell |
|
||||
| clj, edn | Match → Clojure |
|
||||
| cljs, cljc | override → `lexers.Get("Clojure")` |
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
- `internal/processor/languages.go` — MODIFY: add entries to `languageMap` and `GetLanguageName`.
|
||||
- `internal/processor/engine.go` — MODIFY: extend `resolveLexer` overrides.
|
||||
- `internal/processor/engine_test.go` — MODIFY: extend `TestResolveLexer`.
|
||||
- `internal/processor/tier1_test.go` — CREATE: behavior goldens (reuses `runMultilineCase` from `multiline_test.go`).
|
||||
- `internal/processor/testdata/multiline/tier1_*` — CREATE: fixture + `.golden` pairs.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Register languages + lexer routing
|
||||
|
||||
**Files:** `languages.go`, `engine.go`, `engine_test.go`
|
||||
|
||||
- [ ] **Step 1: Extend TestResolveLexer (failing)**
|
||||
|
||||
In `internal/processor/engine_test.go`, add these rows to the `TestResolveLexer` `cases` slice (after the existing rows):
|
||||
|
||||
```go
|
||||
{"a.tf", "Terraform"},
|
||||
{"a.hcl", "HCL"},
|
||||
{"a.tfvars", "Terraform"},
|
||||
{"a.sol", "Solidity"},
|
||||
{"a.zig", "Zig"},
|
||||
{"a.ex", "Elixir"},
|
||||
{"a.exs", "Elixir"},
|
||||
{"a.nix", "Nix"},
|
||||
{"a.proto", "Protocol Buffer"},
|
||||
{"a.graphql", "GraphQL"},
|
||||
{"a.graphqls", "GraphQL"},
|
||||
{"a.ml", "OCaml"},
|
||||
{"a.mli", "OCaml"},
|
||||
{"a.hs", "Haskell"},
|
||||
{"a.clj", "Clojure"},
|
||||
{"a.edn", "Clojure"},
|
||||
{"a.cljs", "Clojure"},
|
||||
{"a.cljc", "Clojure"},
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `cd /home/mroberts/tmp/review/shush-fork && go test ./internal/processor -run TestResolveLexer`
|
||||
Expected: FAIL — `a.tfvars`, `a.cljs`, `a.cljc` resolve to empty (Match misses them).
|
||||
|
||||
- [ ] **Step 3: Extend resolveLexer overrides**
|
||||
|
||||
In `internal/processor/engine.go`, replace the `if ext == "less" || ext == "sass" {` block with a switch:
|
||||
|
||||
```go
|
||||
var l chroma.Lexer
|
||||
switch ext {
|
||||
case "less", "sass":
|
||||
l = lexers.Get("SCSS")
|
||||
case "tfvars":
|
||||
l = lexers.Get("Terraform")
|
||||
case "cljs", "cljc":
|
||||
l = lexers.Get("Clojure")
|
||||
default:
|
||||
l = lexers.Match("f." + ext)
|
||||
}
|
||||
```
|
||||
|
||||
(Keep the surrounding `ext := ...`, the `if l == nil { return nil }`, and `return chroma.Coalesce(l)` unchanged.)
|
||||
|
||||
- [ ] **Step 4: Add languageMap entries**
|
||||
|
||||
In `internal/processor/languages.go`, add these entries to the `languageMap` (place near related groups; exact placement does not matter):
|
||||
|
||||
```go
|
||||
"tf": {LineComment: "#", AlternateLineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||
"hcl": {LineComment: "#", AlternateLineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||
"tfvars": {LineComment: "#", AlternateLineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||
"sol": {LineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||
"zig": {LineComment: "//"},
|
||||
"ex": {LineComment: "#"},
|
||||
"exs": {LineComment: "#"},
|
||||
"nix": {LineComment: "#", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||
"proto": {LineComment: "//", BlockComment: &types.BlockComment{Start: "/*", End: "*/"}},
|
||||
"graphql": {LineComment: "#"},
|
||||
"graphqls": {LineComment: "#"},
|
||||
"ml": {BlockComment: &types.BlockComment{Start: "(*", End: "*)"}},
|
||||
"mli": {BlockComment: &types.BlockComment{Start: "(*", End: "*)"}},
|
||||
"hs": {LineComment: "--", BlockComment: &types.BlockComment{Start: "{-", End: "-}"}},
|
||||
"clj": {LineComment: ";"},
|
||||
"cljs": {LineComment: ";"},
|
||||
"cljc": {LineComment: ";"},
|
||||
"edn": {LineComment: ";"},
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add GetLanguageName entries**
|
||||
|
||||
In `GetLanguageName`'s `names` map add:
|
||||
|
||||
```go
|
||||
"tf": "Terraform",
|
||||
"hcl": "HCL",
|
||||
"tfvars": "Terraform",
|
||||
"sol": "Solidity",
|
||||
"zig": "Zig",
|
||||
"ex": "Elixir",
|
||||
"exs": "Elixir",
|
||||
"nix": "Nix",
|
||||
"proto": "Protocol Buffer",
|
||||
"graphql": "GraphQL",
|
||||
"graphqls": "GraphQL",
|
||||
"ml": "OCaml",
|
||||
"mli": "OCaml",
|
||||
"hs": "Haskell",
|
||||
"clj": "Clojure",
|
||||
"cljs": "ClojureScript",
|
||||
"cljc": "Clojure",
|
||||
"edn": "EDN",
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run resolver test + build**
|
||||
|
||||
Run: `go test ./internal/processor -run TestResolveLexer && go build ./...`
|
||||
Expected: PASS; build clean.
|
||||
|
||||
- [ ] **Step 7: `make check` then commit**
|
||||
|
||||
```bash
|
||||
jj describe -m "feat(languages): register tier-1 languages and chroma routing"
|
||||
jj new
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Behavior goldens
|
||||
|
||||
**Files:** `internal/processor/tier1_test.go` (create), `testdata/multiline/tier1_*` (create)
|
||||
|
||||
**Interfaces consumed:** `runMultilineCase`, `multilineCase` (from `multiline_test.go`, same package).
|
||||
|
||||
- [ ] **Step 1: Write the case table (failing)**
|
||||
|
||||
Create `internal/processor/tier1_test.go`:
|
||||
|
||||
```go
|
||||
package processor
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTier1Languages(t *testing.T) {
|
||||
cases := []multilineCase{
|
||||
{name: "terraform", file: "tier1_terraform.tf"},
|
||||
{name: "solidity", file: "tier1_solidity.sol"},
|
||||
{name: "zig", file: "tier1_zig.zig"},
|
||||
{name: "elixir", file: "tier1_elixir.ex"},
|
||||
{name: "nix", file: "tier1_nix.nix"},
|
||||
{name: "protobuf", file: "tier1_protobuf.proto"},
|
||||
{name: "graphql", file: "tier1_graphql.graphql"},
|
||||
{name: "ocaml", file: "tier1_ocaml.ml"},
|
||||
{name: "haskell", file: "tier1_haskell.hs"},
|
||||
{name: "clojure", file: "tier1_clojure.clj"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
runMultilineCase(t, tc)
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `go test ./internal/processor -run TestTier1Languages`
|
||||
Expected: FAIL — missing `testdata/multiline/tier1_terraform.tf`.
|
||||
|
||||
- [ ] **Step 3: Create fixtures + goldens with exact bytes**
|
||||
|
||||
`tier1_terraform.tf`:
|
||||
```
|
||||
resource "a" "b" { # h
|
||||
x = 1 // s
|
||||
/* block
|
||||
cmt */
|
||||
y = 2
|
||||
}
|
||||
```
|
||||
`tier1_terraform.tf.golden`:
|
||||
```
|
||||
resource "a" "b" {
|
||||
x = 1
|
||||
y = 2
|
||||
}
|
||||
```
|
||||
|
||||
`tier1_solidity.sol`:
|
||||
```
|
||||
contract C {
|
||||
// line
|
||||
/* block
|
||||
cmt */
|
||||
uint x;
|
||||
}
|
||||
```
|
||||
`tier1_solidity.sol.golden`:
|
||||
```
|
||||
contract C {
|
||||
uint x;
|
||||
}
|
||||
```
|
||||
|
||||
`tier1_zig.zig`:
|
||||
```
|
||||
const x = 1; // line
|
||||
// c2
|
||||
pub fn main() void {}
|
||||
```
|
||||
`tier1_zig.zig.golden`:
|
||||
```
|
||||
const x = 1;
|
||||
pub fn main() void {}
|
||||
```
|
||||
|
||||
`tier1_elixir.ex`:
|
||||
```
|
||||
defmodule M do
|
||||
# line
|
||||
def f, do: 1
|
||||
end
|
||||
```
|
||||
`tier1_elixir.ex.golden`:
|
||||
```
|
||||
defmodule M do
|
||||
def f, do: 1
|
||||
end
|
||||
```
|
||||
|
||||
`tier1_nix.nix`:
|
||||
```
|
||||
{
|
||||
# line
|
||||
x = 1;
|
||||
/* block
|
||||
cmt */
|
||||
y = 2;
|
||||
}
|
||||
```
|
||||
`tier1_nix.nix.golden`:
|
||||
```
|
||||
{
|
||||
x = 1;
|
||||
y = 2;
|
||||
}
|
||||
```
|
||||
|
||||
`tier1_protobuf.proto`:
|
||||
```
|
||||
syntax = "proto3";
|
||||
// line
|
||||
message M {
|
||||
/* block
|
||||
cmt */
|
||||
int32 x = 1;
|
||||
}
|
||||
```
|
||||
`tier1_protobuf.proto.golden`:
|
||||
```
|
||||
syntax = "proto3";
|
||||
message M {
|
||||
int32 x = 1;
|
||||
}
|
||||
```
|
||||
|
||||
`tier1_graphql.graphql`:
|
||||
```
|
||||
type Q {
|
||||
# line
|
||||
f: Int
|
||||
}
|
||||
```
|
||||
`tier1_graphql.graphql.golden`:
|
||||
```
|
||||
type Q {
|
||||
f: Int
|
||||
}
|
||||
```
|
||||
|
||||
`tier1_ocaml.ml`:
|
||||
```
|
||||
let x = 1
|
||||
(* block
|
||||
cmt *)
|
||||
let y = 2
|
||||
```
|
||||
`tier1_ocaml.ml.golden`:
|
||||
```
|
||||
let x = 1
|
||||
let y = 2
|
||||
```
|
||||
|
||||
`tier1_haskell.hs`:
|
||||
```
|
||||
main = putStrLn "x" -- line
|
||||
{- block
|
||||
cmt -}
|
||||
f = 1
|
||||
```
|
||||
`tier1_haskell.hs.golden`:
|
||||
```
|
||||
main = putStrLn "x"
|
||||
f = 1
|
||||
```
|
||||
|
||||
`tier1_clojure.clj`:
|
||||
```
|
||||
(ns app) ; line
|
||||
(def x 1)
|
||||
```
|
||||
`tier1_clojure.clj.golden`:
|
||||
```
|
||||
(ns app)
|
||||
(def x 1)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `go test ./internal/processor -run TestTier1Languages -v`
|
||||
Expected: PASS — 10 subtests. If a language fails, STOP and report got/want (a lexer may include/exclude boundary whitespace differently); do not edit the golden to force green.
|
||||
|
||||
- [ ] **Step 5: Full suite + static build**
|
||||
|
||||
Run: `go test ./... && CGO_ENABLED=0 go build -o /tmp/shush_t1 ./cmd/shush && file /tmp/shush_t1`
|
||||
Expected: all green; statically linked.
|
||||
|
||||
- [ ] **Step 6: `make check` then commit**
|
||||
|
||||
```bash
|
||||
jj describe -m "test(languages): tier-1 language comment-removal goldens"
|
||||
jj new
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review (completed during authoring)
|
||||
|
||||
- **Coverage:** 19 extensions registered (Task 1), 11 languages behavior-tested (Task 2). Every extension asserted in `TestResolveLexer`; every language in `TestTier1Languages`.
|
||||
- **Placeholder scan:** exact map entries, exact resolveLexer switch, exact fixture bytes. No TBD.
|
||||
- **Type consistency:** `resolveLexer` switch matches existing signature; `languageMap`/`names` use existing `types.Language`/`types.BlockComment`; `runMultilineCase`/`multilineCase` reused unchanged.
|
||||
- **Comment styles** verified via chroma spike (all 11 languages: lossless + comments removed). `.v` deliberately excluded (ambiguous Coq/V/Verilog).
|
||||
Reference in New Issue
Block a user