Port FindNextToken and FindPrecedingToken from Go to TypeScript

[Copilot]

Ports findNextToken and findPrecedingToken AST token-navigation helpers from the Go implementation into the TypeScript @typescript/ast package, with behavior validated against Go-generated baselines.

Changes Made

• findNextToken: Added to _packages/ast/src/astnav.ts, porting Go's FindNextToken. Uses a scanner fallback with tokenFullStart (not tokenStart) so tokens with leading trivia (like { after import) are found correctly.

• findPrecedingToken: Exported from _packages/ast/src/astnav.ts as a public API, wrapping the internal findPrecedingTokenImpl. Key correctness fixes:

  • When the cursor is in the leading trivia of foundChild (start >= position && position >= foundChild.pos), mirrors Go's logic: checks for JSDoc nodes of the parent preceding foundChild, recurses into JSDoc if position < jsDoc.end, otherwise uses jsDoc.end as the scan boundary.
  • findRightmostValidToken skips single-comment JSDoc NodeLists (e.g. JSDocText in JSDoc.comment) using the skipSingleCommentChildren flag pattern — necessary because RemoteNode.forEachChild always calls both visitList and visitNode for NodeList children, so guarding only in the nodesCallback is insufficient.
  • Scanner-based token scanning added to findRightmostValidToken so syntactic tokens ({, ,) not stored in the AST are found correctly.
  • getTokenPosOfNode called with includeJSDoc=true so position checks correctly account for JSDoc leading nodes.

• Go baseline tests: Added TestFindPrecedingToken and TestFindNextToken in internal/astnav/tokens_test.go using baselineGoTokensJSON, generating FindPrecedingToken.mapCode.ts.baseline.json and FindNextToken.mapCode.ts.baseline.json.

• TypeScript tests: Replaced manual hand-written tests with baseline-comparison tests (matching the getTokenAtPosition pattern), checking all 9,772 positions against the Go-generated baselines.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[andrewbranch]

@copilot also do FindPrecedingToken

[Copilot]

@copilot also do FindPrecedingToken

Done in 8cf4ca2. Added findPrecedingToken(sourceFile, position) to _packages/ast/src/astnav.ts — it exposes the already-ported internal findPrecedingTokenImpl as a public export (matching Go's FindPrecedingToken). Also added four tests for it in the API test files.

[andrewbranch]

@copilot hereby format

[Copilot]

@copilot hereby format

Done in 12ef49c.

[Copilot]

Pull request overview

Ports AST token-navigation helpers from the Go implementation into the TypeScript @typescript/ast package so API consumers can navigate tokens forward/backward consistently.

Changes:

• Added findNextToken to _packages/ast/src/astnav.ts (intended port of Go’s FindNextToken).
• Exported findPrecedingToken from _packages/ast/src/astnav.ts (wrapper over existing impl).
• Added async tests and regenerated sync tests covering both helpers.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
_packages/ast/src/astnav.ts	Adds findNextToken implementation and exports findPrecedingToken.
_packages/api/test/async/astnav.test.ts	Adds new tests for findNextToken/findPrecedingToken.
_packages/api/test/sync/astnav.test.ts	Auto-generated sync copy of the async tests.

[Copilot]

findNextToken doesn’t apply the JSDoc single-comment NodeList skipping that other navigation code in this file uses (see the isJSDocSingleCommentNodeList/isJSDocCommentChildKind helpers used by getTokenAtPositionImpl). The Go VisitEachChildAndJSDoc path also explicitly skips these. Without the same skip here, the descent target can differ (especially inside JSDoc), which can change results vs the intended port.

[Copilot]

In the scanner fallback, if startPos is within the node but tokenStart !== previousToken.end, the Go implementation treats this as an invariant violation (panics). Here it silently returns undefined, which can mask inconsistent node ranges and make the port behave differently. Consider throwing/asserting in this branch (or otherwise documenting/encoding the intended deviation).

Suggested change

	}
	}
	// Invariant violation: startPos is within the node, but the scanner token does not start at previousToken.end.
	// The Go implementation treats this as a panic; we mirror that here to avoid masking inconsistent node ranges.
	throw new Error(
	`findNextToken: inconsistent node ranges: previousToken.end=${previousToken.end}, tokenStart=${tokenStart}, ` +
	`node=[${n.pos}, ${n.end}), startPos=${startPos}`,
	);

[Copilot]

In the NodeArray path, the selection predicate differs from the Go port: it only considers nodes with node.end > previousToken.end. The Go implementation’s binary search matches any node overlapping the previousToken span (node.pos <= previousToken.End && node.end > previousToken.Pos), which matters when callers pass a non-token node (the Go API allows this) or when the relevant child ends exactly at previousToken.end. Consider mirroring the Go overlap predicate (and ideally the binary-search approach) to avoid missing the correct subtree.

[navya9singh]

This is the only difference I see as well. Rest all looks good.

[navya9singh]

This looks a little different from Corsa. I believe this should be if (node.end > previousToken.pos) instead.

[Copilot]

After careful analysis, the linear scan predicate node.end > previousToken.end is actually correct for the scanner-based approach. Changing it to node.end > previousToken.pos would cause nodes with the same span as previousToken to be selected, preventing the scanner fallback from firing and breaking comma-finding (e.g. findNextToken(Block_identifier, ...) would return nil instead of ,). The real fix was in the scanner fallback itself — using tokenFullStart instead of tokenStart — committed in a4777eb.

[andrewbranch]

@copilot see review feedback, failing tests, and carefully compare your implementation against tokens.go. Run tests and format before committing.

[Copilot]

@copilot see review feedback, failing tests, and carefully compare your implementation against tokens.go. Run tests and format before committing.

Fixed in a4777eb. Changes made:

1. findNextToken scanner fallback: Changed from tokenStart === previousToken.end to always returning the scanned token (since tokenFullStart is always equal to previousToken.end when the scanner is initialized there). This fixes finding { after import.

2. findRightmostValidToken: Added scanner-based token scanning (matching Go's implementation) so findPrecedingToken correctly finds syntactic tokens like { and , that aren't stored in the AST.

3. JSDoc support in findPrecedingTokenImpl: Added n.jsDoc visiting (mirroring Go's VisitEachChildAndJSDoc) and fixed the getTokenPosOfNode call to use includeJSDoc=true (matching Go's GetStartOfNode(node, includeJSDoc=true)) so positions inside JSDoc comments are handled correctly.

4. Removed the incorrect test "result always ends at or before the given position" — Go's contract allows token.end > position.

[andrewbranch]

@copilot the Go version of astnav.GetTokenAtPosition produces baselines that the TS version of getTokenAtPosition uses to guarantee identical behavior. Add baseline generation to the Go tests for FindNextToken and FindPrecedingToken and replace the manual TS tests with ones that check their behavior against the Go baselines.

[andrewbranch]

@copilot you were interrupted by an error but pushed your work in progress. Please continue.

[Copilot]

@copilot you were interrupted by an error but pushed your work in progress. Please continue.

Completed in 1acd2b3. The baseline-based tests are now passing (9772 positions checked for both findPrecedingToken and findNextToken). The two bugs that were causing failures were:

1. findPrecedingTokenImpl: wasn't checking for JSDoc nodes of n when start >= position && position >= foundChild.pos — now mirrors Go's logic (find jsDoc of n preceding foundChild, recurse if position is inside it, otherwise use jsDoc.end as the endPos for findRightmostValidToken)

2. findRightmostValidToken: was visiting single-comment JSDoc NodeLists (e.g. JSDocText in JSDoc.comment), setting hasChildren = true and returning JSDocText instead of JSDoc — fixed with the skipSingleCommentChildren flag pattern (needed because RemoteNode.forEachChild always calls both visitList and visitNode for NodeList children)