|
|
1
|
+# GHC Exact Print Annotations — Technical Overview
|
|
|
2
|
+
|
|
|
3
|
+## Motivation
|
|
|
4
|
+
|
|
|
5
|
+A central goal of Haskell tooling — refactoring engines, formatters, language servers, code
|
|
|
6
|
+generators — is to make targeted edits to a Haskell source file and then emit the result with
|
|
|
7
|
+_only the intended changes_ applied. A naive approach of pretty-printing the parsed AST loses
|
|
|
8
|
+all original layout, comments, and stylistic choices, producing diffs that are far larger than
|
|
|
9
|
+the semantic change.
|
|
|
10
|
+
|
|
|
11
|
+GHC's **Exact Print Annotation (EPA)** subsystem solves this by embedding enough positional
|
|
|
12
|
+information directly into every AST node to allow the tree to be reprinted _exactly_ as the
|
|
|
13
|
+original source, byte-for-byte (modulo tab expansion). When a tool modifies the AST it adjusts
|
|
|
14
|
+only the annotations it needs to change; all surrounding nodes reprint themselves unchanged.
|
|
|
15
|
+
|
|
|
16
|
+## Background: the syntax tree
|
|
|
17
|
+
|
|
|
18
|
+The AST of the source program is represented using the data types defined in `Language.Haskell.Syntax.*`.
|
|
|
19
|
+This data type uses the "Trees That Grow (TTG)" plan;
|
|
|
20
|
+see [Implementing trees that grow](https://gitlab.haskell.org/ghc/ghc/-/wikis/implementing-trees-that-grow)
|
|
|
21
|
+
|
|
|
22
|
+For example, in (GHC-independent) `Language.Haskell.Syntax.Expr`:
|
|
|
23
|
+```
|
|
|
24
|
+data HsExpr p
|
|
|
25
|
+ = ... many constructors including ...
|
|
|
26
|
+ | HsLet (XLet p)
|
|
|
27
|
+ (HsLocalBinds p)
|
|
|
28
|
+ (LHsExpr p)
|
|
|
29
|
+
|
|
|
30
|
+type family XLet p
|
|
|
31
|
+type LHsExpr p = XRec p (HsExpr p)
|
|
|
32
|
+
|
|
|
33
|
+```
|
|
|
34
|
+Note that
|
|
|
35
|
+* The `(XLet p)` field is the *extension field* of the `HsLet` data constructor,
|
|
|
36
|
+ where `XLet` is a type family.
|
|
|
37
|
+* Almost every node is wrapped in an `XRec`, another type family. That makes it easy
|
|
|
38
|
+ for clients to attach arbitrary information to each node.
|
|
|
39
|
+
|
|
|
40
|
+GHC specialises this data a type in `GHC.Hs.*`, as follows:
|
|
|
41
|
+```
|
|
|
42
|
+data Pass = Parsed | Renamed | Typechecked
|
|
|
43
|
+
|
|
|
44
|
+data GhcPass (c :: Pass) where
|
|
|
45
|
+ GhcPs :: GhcPass 'Parsed
|
|
|
46
|
+ GhcRn :: GhcPass 'Renamed
|
|
|
47
|
+ GhcTc :: GhcPass 'Typechecked
|
|
|
48
|
+
|
|
|
49
|
+type family XRec p a = r | r -> a
|
|
|
50
|
+type instance XRec (GhcPass p) t = XRecGhc t
|
|
|
51
|
+
|
|
|
52
|
+-- (XRecGhc tree) wraps `tree` in a GHC-specific,
|
|
|
53
|
+-- but pass-independent, source location
|
|
|
54
|
+type XRecGhc t = GenLocated (Anno t) t
|
|
|
55
|
+
|
|
|
56
|
+data GenLocated l e = L l e
|
|
|
57
|
+type family Anno t
|
|
|
58
|
+```
|
|
|
59
|
+So, via `XRec`, every node `e :: t` in the GHC-specific version of HsSyn is wrapped in
|
|
|
60
|
+a `L ann e`, where `ann :: Anno t` is the annotation on the node.
|
|
|
61
|
+
|
|
|
62
|
+Notice that there are two places we can hang information:
|
|
|
63
|
+* (EXT) **Constructor-specific punctuation**: the extension field of each data constructor
|
|
|
64
|
+ can contain information that is specific to that constructor. Example: the location
|
|
|
65
|
+ of the keywords `let` and `in` for the `HsLet` construct.
|
|
|
66
|
+
|
|
|
67
|
+* (XREC) **Entire-node information**: the `Anno t` field that wraps almost every node in the
|
|
|
68
|
+ syntax tree can contain information that is needed for *every* node. Classic example:
|
|
|
69
|
+ the `SrcSpan` of the node.
|
|
|
70
|
+
|
|
|
71
|
+**NOTE**: currently there is information in (XREC) that more properly belongs in (EXT).
|
|
|
72
|
+A refactoring project is under way to put this right.
|
|
|
73
|
+
|
|
|
74
|
+
|
|
|
75
|
+## File layout
|
|
|
76
|
+
|
|
|
77
|
+The subsystem lives primarily in `utils/check-exact/` and is exposed through three main layers:
|
|
|
78
|
+
|
|
|
79
|
+| Layer | Key operation | Purpose |
|
|
|
80
|
+|---|---|---|
|
|
|
81
|
+| **Parser** | `getCommentsFor` / `getPriorCommentsFor` | Attaches comments to AST nodes |
|
|
|
82
|
+| **Printer** | `exactPrint` / `makeDeltaAst` | Reprints the AST; converts absolute spans to relative deltas |
|
|
|
83
|
+| **Transform** | `Transform` monad | Edits the AST while keeping annotations consistent |
|
|
|
84
|
+
|
|
|
85
|
+---
|
|
|
86
|
+
|
|
|
87
|
+## Core Concepts
|
|
|
88
|
+
|
|
|
89
|
+### Positions: Absolute and Relative
|
|
|
90
|
+
|
|
|
91
|
+The parser records every token's position as an **absolute** source span (file, line, column).
|
|
|
92
|
+Before reprinting a modified AST the printer converts these to **relative** (delta) form.
|
|
|
93
|
+
|
|
|
94
|
+```
|
|
|
95
|
+EpaLocation
|
|
|
96
|
+ ├─ EpaSpan (absolute) — original (line, col) span from the parser
|
|
|
97
|
+ └─ EpaDelta (relative) — DeltaPos + retained original span + leading comments
|
|
|
98
|
+```
|
|
|
99
|
+
|
|
|
100
|
+A `DeltaPos` encodes a position as an offset from a _reference point_:
|
|
|
101
|
+
|
|
|
102
|
+```
|
|
|
103
|
+DeltaPos
|
|
|
104
|
+ ├─ SameLine col — rightward gap from the end of the preceding token; col ≥ 0
|
|
|
105
|
+ └─ DifferentLine lines col
|
|
|
106
|
+ — lines below the preceding token;
|
|
|
107
|
+ col is a SIGNED offset from the enclosing layout block's
|
|
|
108
|
+ left margin (layout_lhs + col = absolute column)
|
|
|
109
|
+```
|
|
|
110
|
+
|
|
|
111
|
+The different reference points for the two constructors are deliberate and are what makes
|
|
|
112
|
+layout-preserving moves possible (see _Splice Invariance_ below).
|
|
|
113
|
+
|
|
|
114
|
+---
|
|
|
115
|
+
|
|
|
116
|
+## Invariants
|
|
|
117
|
+
|
|
|
118
|
+### 1. Every Node Is Self-Contained
|
|
|
119
|
+
|
|
|
120
|
+Each located AST node carries a `NodeAnnotation` (the `EpAnn ann` wrapper) containing:
|
|
|
121
|
+
|
|
|
122
|
+- **`anchor`** — an `EpaLocation` marking the top-left corner of the node's bounding rectangle.
|
|
|
123
|
+ This is the coordinate origin for the node's internal layout.
|
|
|
124
|
+- **`anns`** — node-kind-specific annotation payload (keyword token positions, bracket tokens,
|
|
|
125
|
+ pragma text, etc.).
|
|
|
126
|
+- **`comments`** — all `EpaComment` values logically owned by this node.
|
|
|
127
|
+
|
|
|
128
|
+No separate annotation map exists. The annotation is embedded directly in the `GenLocated`
|
|
|
129
|
+wrapper via the `Anno` type family (the "Trees That Grow" mechanism). The annotation type for
|
|
|
130
|
+each node kind is determined statically by its Haskell type.
|
|
|
131
|
+
|
|
|
132
|
+### 2. Trailing Annotations Are Separate from the Bounding Box
|
|
|
133
|
+
|
|
|
134
|
+Punctuation that separates list elements — trailing commas, semicolons, vertical bars, and
|
|
|
135
|
+constraint arrows — is stored in a `TrailingAnn` list that is _outside_ the node's bounding box.
|
|
|
136
|
+
|
|
|
137
|
+```
|
|
|
138
|
+TrailingAnn
|
|
|
139
|
+ ├─ semi — ';'
|
|
|
140
|
+ ├─ comma — ','
|
|
|
141
|
+ ├─ vbar — '|'
|
|
|
142
|
+ └─ darrow — '=>' or '⇒'
|
|
|
143
|
+```
|
|
|
144
|
+
|
|
|
145
|
+The `HasTrailing` typeclass exposes a uniform `trailing` / `setTrailing` interface over all
|
|
|
146
|
+annotation types. The printer uses this interface to extract trailing glue without inspecting
|
|
|
147
|
+the concrete annotation type.
|
|
|
148
|
+
|
|
|
149
|
+**Consequence for tooling:** when moving, copying, or deleting a node, a tool operates on the
|
|
|
150
|
+anchor span and can transfer or discard trailing annotations independently. There is no risk
|
|
|
151
|
+of accidentally duplicating or losing a separator when restructuring a list.
|
|
|
152
|
+
|
|
|
153
|
+### 3. The Bounding Box Invariant
|
|
|
154
|
+
|
|
|
155
|
+The `anchor` span is the bounding box of everything the node _owns_:
|
|
|
156
|
+
|
|
|
157
|
+- **Leading comments** are printed _before_ the anchor and lie _outside_ the bounding box.
|
|
|
158
|
+ They may appear at any indentation.
|
|
|
159
|
+- **Trailing annotations** are printed _after_ the anchor's end and also lie outside it.
|
|
|
160
|
+- The anchor's start position is the top-left corner of the box; the parser widens the span
|
|
|
161
|
+ (via `widenAnchorT` / `widenAnchorS`) to cover every token that syntactically belongs to
|
|
|
162
|
+ the node.
|
|
|
163
|
+
|
|
|
164
|
+```
|
|
|
165
|
+ leading comment ← outside bounding box; before anchor
|
|
|
166
|
+ [anchor start]
|
|
|
167
|
+ token token token ← inside bounding box
|
|
|
168
|
+ token token
|
|
|
169
|
+ [anchor end]
|
|
|
170
|
+ , ← trailing annotation; outside bounding box
|
|
|
171
|
+```
|
|
|
172
|
+
|
|
|
173
|
+### 4. Splice Invariance
|
|
|
174
|
+
|
|
|
175
|
+Moving an entire subtree to a different column requires updating _only the root node's entry
|
|
|
176
|
+delta_. No descendant delta needs to change.
|
|
|
177
|
+
|
|
|
178
|
+This works because `DifferentLine` deltas store the column as `absolute_col − layout_lhs`, where
|
|
|
179
|
+`layout_lhs` is the left margin of the enclosing layout block. When the root anchor moves, the
|
|
|
180
|
+printer adopts the new column as the new `layout_lhs`. Every child that has a `DifferentLine`
|
|
|
181
|
+delta recomputes its absolute column as `new_layout_lhs + stored_col`, sliding the entire subtree
|
|
|
182
|
+to the right or left by the same amount.
|
|
|
183
|
+
|
|
|
184
|
+`SameLine` deltas are unaffected by relocation because they record a gap from the immediately
|
|
|
185
|
+preceding token, not from the layout margin.
|
|
|
186
|
+
|
|
|
187
|
+**Known exceptions:** layout-block terminators such as `in`, `where`, and `of` intentionally
|
|
|
188
|
+appear to the _left_ of the block they close. Their `DifferentLine.col` is negative. The
|
|
|
189
|
+printer's validity check (`isGoodDelta`) accepts this: for `DifferentLine` it only requires
|
|
|
190
|
+`lines > 0`, not that `col ≥ 0`.
|
|
|
191
|
+
|
|
|
192
|
+### 5. The Zero-Column Invariant for List Items
|
|
|
193
|
+
|
|
|
194
|
+Nodes that appear as direct children of a layout list (a `where` clause, a `do` block, an
|
|
|
195
|
+export list, etc.) always store `DifferentLine(n, 0)` — a zero column offset from the list's
|
|
|
196
|
+layout origin.
|
|
|
197
|
+
|
|
|
198
|
+The list's layout origin is recorded in `AnnList.al_anchor`. The absolute column of each item
|
|
|
199
|
+is therefore `al_anchor.col + 0 = al_anchor.col`. This has two consequences:
|
|
|
200
|
+
|
|
|
201
|
+1. **Reordering is free.** Items can be moved within the list without updating any individual
|
|
|
202
|
+ delta; all items already share the same column.
|
|
|
203
|
+2. **Relocation is cheap.** Moving the entire list to a new indentation requires updating only
|
|
|
204
|
+ `al_anchor`; all items self-correct automatically.
|
|
|
205
|
+
|
|
|
206
|
+---
|
|
|
207
|
+
|
|
|
208
|
+## Data Flow
|
|
|
209
|
+
|
|
|
210
|
+```
|
|
|
211
|
+Source file
|
|
|
212
|
+ │
|
|
|
213
|
+ ▼
|
|
|
214
|
+┌─────────────┐
|
|
|
215
|
+│ Parser │ getCommentsFor / getPriorCommentsFor / getFinalCommentsFor
|
|
|
216
|
+│ (GHC.Parser│ → allocates EpaComments to each AST node (unbalanced form)
|
|
|
217
|
+│ + Lexer) │ → all EpaLocations are absolute (EpaSpan)
|
|
|
218
|
+└──────┬──────┘
|
|
|
219
|
+ │ ParsedSource (absolute annotations)
|
|
|
220
|
+ ▼
|
|
|
221
|
+┌──────────────────┐
|
|
|
222
|
+│ makeDeltaAst │ converts EpaSpan → EpaDelta for every can_update node
|
|
|
223
|
+│ (exactPrint.hs) │ adjustDeltaForOffset stores col as (absolute - layout_lhs)
|
|
|
224
|
+└──────┬───────────┘
|
|
|
225
|
+ │ ParsedSource (relative annotations)
|
|
|
226
|
+ ▼
|
|
|
227
|
+┌──────────────────────┐
|
|
|
228
|
+│ Transform monad │ structural edits: insert/remove/reorder declarations
|
|
|
229
|
+│ (Transform.hs) │ setEntryDP, transferEntryDP, balanceCommentsList, …
|
|
|
230
|
+└──────┬───────────────┘
|
|
|
231
|
+ │ modified ParsedSource
|
|
|
232
|
+ ▼
|
|
|
233
|
+┌──────────────────┐
|
|
|
234
|
+│ exactPrint │ single-pass traversal; undelta(prior_end, dp, layout_lhs)
|
|
|
235
|
+│ (ExactPrint.hs) │ comment interleaving; layout block tracking
|
|
|
236
|
+└──────┬───────────┘
|
|
|
237
|
+ │
|
|
|
238
|
+ ▼
|
|
|
239
|
+ Source text (byte-for-byte faithful to the original, modulo edits)
|
|
|
240
|
+```
|
|
|
241
|
+
|
|
|
242
|
+---
|
|
|
243
|
+
|
|
|
244
|
+## The Parser Phase
|
|
|
245
|
+
|
|
|
246
|
+### Comment Allocation
|
|
|
247
|
+
|
|
|
248
|
+The GHC parser maintains a comment queue (`comment_q`) of `EpaComment` values seen in the token
|
|
|
249
|
+stream but not yet attached to any node. As each grammar production is reduced, the parser drains
|
|
|
250
|
+matching comments from the queue:
|
|
|
251
|
+
|
|
|
252
|
+- `getCommentsFor span node` — moves all comments whose span falls _inside_ `span` into the
|
|
|
253
|
+ node's `EpAnn.comments.prior` list.
|
|
|
254
|
+- `getPriorCommentsFor span node` — additionally captures comments that fall _just before_ the
|
|
|
255
|
+ span (used for top-level declarations to capture preceding Haddock comments).
|
|
|
256
|
+- `getFinalCommentsFor module_node` — at EOF, drains the entire queue into the module node.
|
|
|
257
|
+
|
|
|
258
|
+**Guarantee:** each comment is allocated to exactly one node; the queue is drained monotonically.
|
|
|
259
|
+All `EpaComments` produced by the parser are in the `unbalanced` state (only `prior`, no
|
|
|
260
|
+`following`).
|
|
|
261
|
+
|
|
|
262
|
+### `Anno` Type Family
|
|
|
263
|
+
|
|
|
264
|
+The annotation type for each AST node kind is chosen by the `Anno` type family. Common
|
|
|
265
|
+specialisations:
|
|
|
266
|
+
|
|
|
267
|
+| Type alias | Annotation payload | Used for |
|
|
|
268
|
+|---|---|---|
|
|
|
269
|
+| `SrcSpanAnnA` | `AnnListItem` | Most expressions, patterns, declarations |
|
|
|
270
|
+| `SrcSpanAnnN` | `NameAnn` | Name occurrences |
|
|
|
271
|
+| `SrcSpanAnnL` | `AnnList ()` | Plain layout lists |
|
|
|
272
|
+| `SrcSpanAnnLW` | `AnnList EpToken` | `where`-clause lists |
|
|
|
273
|
+| `SrcSpanAnnP` | `AnnPragma` | `{-# … #-}` pragmas |
|
|
|
274
|
+| `SrcSpanAnnC` | `AnnContext` | Constraint contexts `C a =>` |
|
|
|
275
|
+
|
|
|
276
|
+---
|
|
|
277
|
+
|
|
|
278
|
+## The Printing Phase
|
|
|
279
|
+
|
|
|
280
|
+### The `ExactPrint` Typeclass
|
|
|
281
|
+
|
|
|
282
|
+Every AST constructor that can appear in a located position has an `ExactPrint` instance:
|
|
|
283
|
+
|
|
|
284
|
+```haskell
|
|
|
285
|
+class (Typeable a) => ExactPrint a where
|
|
|
286
|
+ getAnnotationEntry :: a -> Entry -- extract anchor + trailing + comments
|
|
|
287
|
+ setAnnotationAnchor :: a -> EpaLocation
|
|
|
288
|
+ -> [TrailingAnn] -> EpAnnComments -> a -- write anchor back
|
|
|
289
|
+ exact :: (Monad m, Monoid w) => a -> EP w m a -- print this constructor
|
|
|
290
|
+```
|
|
|
291
|
+
|
|
|
292
|
+`markAnnotated a = enterAnn (getAnnotationEntry a) a` is the single recursive call site.
|
|
|
293
|
+`enterAnn` handles all the cross-cutting concerns (comments, layout, anchor write-back) and
|
|
|
294
|
+then delegates to `exact` for the constructor-specific tokens and child traversals. There are
|
|
|
295
|
+several hundred `ExactPrint` instances, one for each GHC AST constructor.
|
|
|
296
|
+
|
|
|
297
|
+The same instance serves both `exactPrint` and `makeDeltaAst`: the output options (`EPOptions`)
|
|
|
298
|
+determine whether tokens are accumulated into a string or discarded.
|
|
|
299
|
+
|
|
|
300
|
+### `makeDeltaAst`
|
|
|
301
|
+
|
|
|
302
|
+Converts every `EpaSpan` anchor to `EpaDelta` form in a single pass over the AST. The delta
|
|
|
303
|
+is computed as:
|
|
|
304
|
+
|
|
|
305
|
+```
|
|
|
306
|
+raw_delta = ss2delta(prior_end_pos, anchor_start)
|
|
|
307
|
+stored_delta = adjustDeltaForOffset(layout_lhs, raw_delta)
|
|
|
308
|
+ = DifferentLine(lines, absolute_col - layout_lhs)
|
|
|
309
|
+```
|
|
|
310
|
+
|
|
|
311
|
+The inverse during printing (`undelta`) recovers the absolute column:
|
|
|
312
|
+
|
|
|
313
|
+```
|
|
|
314
|
+absolute_col = layout_lhs + stored_delta.col -- DifferentLine
|
|
|
315
|
+absolute_col = prior_col + stored_delta.col -- SameLine
|
|
|
316
|
+```
|
|
|
317
|
+
|
|
|
318
|
+This shared traversal runs with no-op output options; no source text is accumulated.
|
|
|
319
|
+
|
|
|
320
|
+### `exactPrint`
|
|
|
321
|
+
|
|
|
322
|
+Runs the same traversal as `makeDeltaAst` but with string-accumulating options. For each node:
|
|
|
323
|
+
|
|
|
324
|
+1. Leading comments (in `EpaDelta.leading_comments`) are printed before the anchor.
|
|
|
325
|
+2. The anchor position is resolved (absolute from `EpaSpan`, or via `undelta` for `EpaDelta`).
|
|
|
326
|
+3. If `mark_layout` is set, the anchor column becomes the new `layout_lhs` for the block.
|
|
|
327
|
+4. Children are visited recursively with the updated layout state.
|
|
|
328
|
+
|
|
|
329
|
+### Comment Interleaving
|
|
|
330
|
+
|
|
|
331
|
+Two strategies are used depending on the anchor form:
|
|
|
332
|
+
|
|
|
333
|
+- **Absolute anchor (`EpaSpan`):** the printer maintains a pool of pending comments sorted by
|
|
|
334
|
+ source position. Before each token it emits all pooled comments whose position precedes the
|
|
|
335
|
+ token (`printCommentsBefore`, via `commentAllocationBefore`).
|
|
|
336
|
+- **Relative anchor (`EpaDelta`):** comments are pre-attached to the node's annotation by
|
|
|
337
|
+ `makeDeltaAst` or `balanceCommentsList`. The entire pool is flushed unconditionally via
|
|
|
338
|
+ `flushComments`, then the node's own attached comments are printed in order via `printOneComment`.
|
|
|
339
|
+
|
|
|
340
|
+CPP-injected comments may carry fake filenames ("CPP", "LINE", "SHEBANG") in their spans.
|
|
|
341
|
+Ordering always uses `ss2pos` (line, column only) rather than the full `RealSrcSpan` comparator,
|
|
|
342
|
+which would sort by filename first and produce incorrect relative ordering.
|
|
|
343
|
+
|
|
|
344
|
+### Parentheses in Infix Declaration Heads via the Comment Machinery
|
|
|
345
|
+
|
|
|
346
|
+Infix type-level declarations — type synonyms, data types, class declarations, type families, and
|
|
|
347
|
+GADT constructors — may carry optional parentheses around the operator/constructor in the
|
|
|
348
|
+declaration head. For example:
|
|
|
349
|
+
|
|
|
350
|
+```haskell
|
|
|
351
|
+type (a `MyOp` b) = ...
|
|
|
352
|
+class (a `MyClass` b) where ...
|
|
|
353
|
+data (a `MyData` b) = ...
|
|
|
354
|
+```
|
|
|
355
|
+
|
|
|
356
|
+These parentheses are stored as _lists_ of `EpToken` values (`ops`, `cps`) on the declaration's
|
|
|
357
|
+annotation (`AnnSynDecl`, `AnnClassDecl`, `AnnFamilyDecl`, `AnnDataDefn`, `AnnConDeclGADT`, and
|
|
|
358
|
+`AnnFunRhs` for infix function patterns). They cannot be emitted at a fixed point in the
|
|
|
359
|
+structural traversal because the infix head visits the type constructor and its arguments in an
|
|
|
360
|
+order that does not naturally interleave with the surrounding parens.
|
|
|
361
|
+
|
|
|
362
|
+The solution is `epTokensToComments "(" ops` / `epTokensToComments ")" cps`: each present
|
|
|
363
|
+`EpToken` is converted to a synthetic `Comment` (with `keyword_origin = "("` or `")"`) and
|
|
|
364
|
+injected into the comment pool. The pool is ordered by source position, so these synthetic
|
|
|
365
|
+parens are automatically emitted at exactly the right location relative to the other tokens,
|
|
|
366
|
+without requiring any explicit position tracking in the `exact` instance.
|
|
|
367
|
+
|
|
|
368
|
+During `makeDeltaAst`, each synthetic paren's computed delta is captured via `applyComment` and
|
|
|
369
|
+embedded in the adjacent token's `EpaDelta` constructor, so the round-trip preserves paren
|
|
|
370
|
+positions in delta form too.
|
|
|
371
|
+
|
|
|
372
|
+### Mixed Bind/Sig Lists
|
|
|
373
|
+
|
|
|
374
|
+The GHC AST stores function bindings and type signatures in separate sub-lists. The printer
|
|
|
375
|
+merges them using a sort key on the `ValBinds` extension point:
|
|
|
376
|
+
|
|
|
377
|
+- **`NoAnnSortKey` (span order):** sorts by original source span — correct immediately after
|
|
|
378
|
+ parsing while spans are still trustworthy.
|
|
|
379
|
+- **`AnnSortKey [BindTag]` (tag order):** a sequence of `bind_tag` / `sig_tag` tokens recording
|
|
|
380
|
+ the exact interleaving. Set whenever `replaceDeclarations` is called; the printer follows the
|
|
|
381
|
+ sequence exactly, enabling caller-controlled reordering.
|
|
|
382
|
+
|
|
|
383
|
+Class and instance bodies use the same mechanism with four sub-lists and `DeclTag`.
|
|
|
384
|
+
|
|
|
385
|
+---
|
|
|
386
|
+
|
|
|
387
|
+## The Transform Phase
|
|
|
388
|
+
|
|
|
389
|
+The `Transform` monad wraps a state carrying a unique-span counter and a debug log. It exposes
|
|
|
390
|
+operations for structural edits while keeping annotations consistent.
|
|
|
391
|
+
|
|
|
392
|
+The unique-span counter exists to support `uniqueSrcSpanT`, which allocated synthetic `SrcSpan`
|
|
|
393
|
+values (line = -1) for freshly created AST nodes. That function is now dead code: it is exported
|
|
|
394
|
+for API compatibility but is called nowhere in GHC or `check-exact`. New nodes are given
|
|
|
395
|
+`EpaDelta` locations directly, which need no unique span. The counter field in `TransformState`
|
|
|
396
|
+is therefore vestigial and should be removed together with `uniqueSrcSpanT` and `isUniqueSrcSpan`
|
|
|
397
|
+in a future cleanup.
|
|
|
398
|
+
|
|
|
399
|
+### Declaration Access
|
|
|
400
|
+
|
|
|
401
|
+- **`getDeclarations node`** — returns the declaration list of a module, match, let-binding,
|
|
|
402
|
+ pattern-binding where clause, or class/instance body as a `DeclarationList`.
|
|
|
403
|
+- **`replaceDeclarations node new_decls`** — replaces the list; implicitly records the new
|
|
|
404
|
+ ordering as a tag-order `AnnSortKey`, so the printer will honour the caller's order regardless
|
|
|
405
|
+ of original source spans. **Passing an empty list collapses the underlying `HsLocalBinds` to
|
|
|
406
|
+ `EmptyLocalBinds`, which drops the `where` keyword annotation entirely** — this is not a no-op
|
|
|
407
|
+ on the container structure.
|
|
|
408
|
+
|
|
|
409
|
+A key responsibility of `getDeclarations` is **normalising `ValBinds`**. Inside a
|
|
|
410
|
+`HsLocalBinds` / `ValBinds` node (a `where` clause, `let` expression, `let` statement, or
|
|
|
411
|
+pattern-binding where clause), GHC stores function bindings and type signatures in two _separate_
|
|
|
412
|
+sub-collections: a `Bag` of `LHsBind` and a `[LSig]`. These are physically disjoint from each
|
|
|
413
|
+other and have no guaranteed ordering relative to each other.
|
|
|
414
|
+
|
|
|
415
|
+`getDeclarations` (via `hsDeclsLocalBinds` → `hsDeclsValBinds` → `orderedDeclsBinds`) merges
|
|
|
416
|
+them back into a single `[LHsDecl]` in source order:
|
|
|
417
|
+
|
|
|
418
|
+- **`NoAnnSortKey` (freshly parsed):** sorts the combined list by `RealSrcSpan`, recovering the
|
|
|
419
|
+ original interleaved order from the absolute source positions.
|
|
|
420
|
+- **`AnnSortKey [BindTag]` (after any structural edit):** replays the tag sequence recorded by
|
|
|
421
|
+ `captureOrderBinds`, drawing from the binds bag or sigs list according to each tag.
|
|
|
422
|
+
|
|
|
423
|
+The result is a physically ordered, uniform list of `LHsDecl` values — binds wrapped in `ValD`,
|
|
|
424
|
+sigs wrapped in `SigD` — that the exact-print and transformation machinery can traverse and edit
|
|
|
425
|
+without knowing or caring that the underlying storage splits them across two separate containers.
|
|
|
426
|
+`replaceDeclarations` is the inverse: it re-separates the uniform list back into the two
|
|
|
427
|
+sub-collections via `decl2Bind` / `decl2Sig` and records a fresh `AnnSortKey` for the new order.
|
|
|
428
|
+
|
|
|
429
|
+### Positioning Nodes
|
|
|
430
|
+
|
|
|
431
|
+- **`setEntryDP node dp`** — sets the entry delta of a node, converting its anchor to relative
|
|
|
432
|
+ form. Must be called before inserting a freshly constructed node.
|
|
|
433
|
+- **`getEntryDP node`** — reads the current entry delta (default `SameLine 0` if absent).
|
|
|
434
|
+- **`transferEntryDP source target`** — moves the entry delta _and_ leading comments from one
|
|
|
435
|
+ node to another. Used when a removed declaration's successor needs to inherit its spacing.
|
|
|
436
|
+
|
|
|
437
|
+### Inserting and Removing Declarations
|
|
|
438
|
+
|
|
|
439
|
+**`insertDeclaration node decl point`** inserts `decl` at `point` (start, end, before/after a
|
|
|
440
|
+named item). The inserted node must already have a relative anchor. All existing declarations
|
|
|
441
|
+are comment-balanced before the structural change.
|
|
|
442
|
+
|
|
|
443
|
+**`removeDeclaration node decl`** removes `decl`. The caller must then call `transferEntryDP`
|
|
|
444
|
+or `setEntryDP` on the new first declaration to absorb the gap.
|
|
|
445
|
+
|
|
|
446
|
+### Comment Balancing
|
|
|
447
|
+
|
|
|
448
|
+After parsing, all comments are in the `prior` list of the nodes that saw them first. The
|
|
|
449
|
+`balanceComments` / `balanceCommentsList` operations redistribute them:
|
|
|
450
|
+
|
|
|
451
|
+- A comment ≤ 1 blank line below declaration D stays with D as a `following` comment.
|
|
|
452
|
+- A comment > 1 blank line below D migrates to the `prior` list of D's successor.
|
|
|
453
|
+
|
|
|
454
|
+The operation is **idempotent**: running it twice produces the same result because `balanced` is
|
|
|
455
|
+a terminal state (`unbalanced → balanced`; no transition out of `balanced` is defined).
|
|
|
456
|
+
|
|
|
457
|
+For `FunBind` nodes the balancing is hierarchical: comments are first balanced at the binding
|
|
|
458
|
+level, then distributed among the individual match equations within the binding.
|
|
|
459
|
+
|
|
|
460
|
+### Capturing Spacing Before Edits
|
|
|
461
|
+
|
|
|
462
|
+Before removing or reordering declarations, spacing must be encoded in relative form so it
|
|
|
463
|
+survives the structural change. If `makeDeltaAst` has **not** been called, the AST still
|
|
|
464
|
+carries absolute `EpaSpan` anchors; the spacing functions below derive relative deltas from those
|
|
|
465
|
+absolute positions and write them back, making subsequent structural edits safe. If `makeDeltaAst`
|
|
|
466
|
+**has** already been called, the anchors are already in `EpaDelta` form and these functions are
|
|
|
467
|
+idempotent — they recompute the same delta values that `makeDeltaAst` already stored.
|
|
|
468
|
+
|
|
|
469
|
+- **`captureLineSpacing decls`** — sets each declaration's entry delta lines to the actual blank
|
|
|
470
|
+ lines between it and its predecessor.
|
|
|
471
|
+- **`captureMatchLineSpacing decl`** — same, within the match equations of a `FunBind`.
|
|
|
472
|
+- **`captureTypeSigSpacing sig`** — captures spacing within a multi-name type signature.
|
|
|
473
|
+- **`addModuleCommentOrigDeltas module`** — converts module-level comment spans from absolute to
|
|
|
474
|
+ relative form before the module's declaration list is replaced.
|
|
|
475
|
+
|
|
|
476
|
+---
|
|
|
477
|
+
|
|
|
478
|
+## CPP Support
|
|
|
479
|
+
|
|
|
480
|
+Source files using `{-# LANGUAGE CPP #-}` present a special challenge: the GHC parser sees
|
|
|
481
|
+the _preprocessed_ text, which may differ substantially from the original source (directives
|
|
|
482
|
+removed, macros expanded).
|
|
|
483
|
+
|
|
|
484
|
+### Pipeline
|
|
|
485
|
+
|
|
|
486
|
+```
|
|
|
487
|
+Original source
|
|
|
488
|
+ │
|
|
|
489
|
+ ├─ getPreprocessorAsComments ─→ directive line tokens (real filename)
|
|
|
490
|
+ │
|
|
|
491
|
+ ├─ stripLinePragmas / tokeniseOriginalSrc ─→ directive-stripped token stream
|
|
|
492
|
+ │
|
|
|
493
|
+ └─ getPreprocessedSrcDirect ─→ C-preprocessed text
|
|
|
494
|
+ └─ lexTokenStream ─→ post-CPP token stream
|
|
|
495
|
+ │
|
|
|
496
|
+ └─ getCppTokens ─→ three-way merge
|
|
|
497
|
+ ─→ MergedCppComments
|
|
|
498
|
+```
|
|
|
499
|
+
|
|
|
500
|
+`getCppTokens` identifies directive tokens present in the original source but absent from the
|
|
|
501
|
+preprocessed output (consumed by CPP) and converts them to synthetic `ITlineComment` tokens.
|
|
|
502
|
+These are merged with the directive lines extracted directly to form `injected_comments`.
|
|
|
503
|
+
|
|
|
504
|
+**Limitation:** the three-way merge is correct only in `-nomacro` mode. Macro expansion
|
|
|
505
|
+introduces expanded tokens that are not recoverable by span-based matching.
|
|
|
506
|
+
|
|
|
507
|
+### Insertion
|
|
|
508
|
+
|
|
|
509
|
+`insertCppComments module injected_comments` splices the synthetic comments back into the parsed
|
|
|
510
|
+AST using a bottom-up traversal (`everywhereM`): each `EpAnn` node claims the injected comments
|
|
|
511
|
+whose source span it encloses. Remaining comments are distributed to module-level positions by
|
|
|
512
|
+`insertTopLevelCppComments`. CPP-aware ordering (`ss2pos`, ignoring filenames) is used
|
|
|
513
|
+throughout.
|
|
|
514
|
+
|
|
|
515
|
+After `insertCppComments`, the round-trip guarantee extends to CPP-enabled sources:
|
|
|
516
|
+`exactPrint(insertCppComments(parseModuleEpAnnsWithCpp(f)))` reproduces `f` byte-for-byte.
|
|
|
517
|
+
|
|
|
518
|
+---
|
|
|
519
|
+
|
|
|
520
|
+## Correctness Guarantees
|
|
|
521
|
+
|
|
|
522
|
+| Guarantee | Statement |
|
|
|
523
|
+|---|---|
|
|
|
524
|
+| **RoundTripFidelity** | For any file `f` parsed by GHC, `exactPrint(makeDeltaAst(parse(f)))` is byte-identical to `f` (modulo tab expansion). |
|
|
|
525
|
+| **AllCommentsEmitted** | After a complete `exactPrint` traversal, every comment attached to the AST has been emitted exactly once and the pending pool is empty. |
|
|
|
526
|
+| **CursorNonDecreasing** | The output cursor only moves forward; tokens are emitted in source order. |
|
|
|
527
|
+| **CommentsAllocatedAtMostOnce** | Each comment in the parser's queue is moved to exactly one AST node; once removed it is never re-added. |
|
|
|
528
|
+| **BalanceIsIdempotent** | `balanceCommentsList` can be called multiple times on the same list without duplicating or losing comments. |
|
|
|
529
|
+| **ReplacePreservesCallerOrder** | After `replaceDeclarations(node, new_decls)`, the printer outputs declarations in the same order as `new_decls`. |
|
|
|
530
|
+| **RoundTripWithCpp** | For CPP-enabled files, `exactPrint(insertCppComments(parseModuleEpAnnsWithCpp(f)))` reproduces `f` byte-for-byte (no-macro mode only). |
|
|
|
531
|
+
|
|
|
532
|
+---
|
|
|
533
|
+
|
|
|
534
|
+## Typical Tool Workflow
|
|
|
535
|
+
|
|
|
536
|
+A refactoring tool that wants to add a new top-level declaration does the following:
|
|
|
537
|
+
|
|
|
538
|
+```
|
|
|
539
|
+1. parseModuleEpAnnsWithCpp opts libdir file -- parse with CPP support
|
|
|
540
|
+2. insertCppComments parsed_source comments -- re-insert CPP/LINE/SHEBANG comments
|
|
|
541
|
+3. makeDeltaAst parsed_source -- convert absolute spans to deltas
|
|
|
542
|
+4. runTransform do
|
|
|
543
|
+ addModuleCommentOrigDeltas module -- protect module-level comments
|
|
|
544
|
+ decls ← getDeclarations module -- read current declaration list
|
|
|
545
|
+ captureLineSpacing decls -- encode spacing before edits
|
|
|
546
|
+ balanceCommentsList decls -- redistribute comments
|
|
|
547
|
+ setEntryDP new_decl (DifferentLine 2 0) -- position the new declaration
|
|
|
548
|
+ replaceDeclarations module (decls ++ [new_decl])
|
|
|
549
|
+5. exactPrint modified_source -- emit the result
|
|
|
550
|
+```
|
|
|
551
|
+
|
|
|
552
|
+The result is the original source with the new declaration appended, all original comments and
|
|
|
553
|
+layout preserved, and a diff that contains only the added lines. |