Add SAS language support

[Allan Bowe (allanbowe)]

Closes #2613 and implemented in https://github.com/sasjs/server

Summary

Adds a Monarch grammar and language configuration for SAS. The issue has been open since 2021, and the maintainer's position there was:

SAS is on place 21 on the TIOBE index, so I think its relevance definitely justifies including a grammar for it. We are very open for a PR on this!

This is tokenization only. It registers no language server and adds no runtime dependency.

Files

file	purpose
src/languages/definitions/sas/sas.ts	the grammar and language configuration
src/languages/definitions/sas/register.ts	registerLanguage for id sas, extension .sas
src/languages/definitions/sas/keywords.js	regenerates the word lists in sas.ts
src/languages/definitions/sas/sas.test.ts	tokenization tests
src/languages/definitions/register.all.ts	one import line
website/src/website/data/home-samples/sample.sas.txt	required by build/check-samples

Design: keywords match in statement position only

SAS is case-insensitive and its statements are bare words, so a grammar that matches keywords anywhere mis-colours ordinary variables. x, means, lib, output and missing are each both a statement and a very common variable name, and x = 1 is far more common in real SAS code than the X statement.

The grammar therefore keeps two states:

• root is statement position. The first word of a statement resolves against the statement list, so data, proc, run, if and x 'dir'; are keywords.
• body is the inside of a statement. Here a bare word resolves against options, control keywords and function names, and a statement keyword is no longer special. if x then output; gives x plain, then and output as control keywords, while output = 1; leaves output plain.

An assignment target is excluded by a lookahead ([a-z_]\w*(?=\s*=)), which is what keeps x = 1 and output = 1 uncoloured without losing the X and OUTPUT statements.

Separate states cover the parts of the language with their own vocabulary:

• sql for PROC SQL and PROC FEDSQL, entered by proc sql and left by quit or run
• ods and options, which carry tagset, style and system option lists
• datalines and datalines4, which render instream data as a string until the terminating semicolon
• double-quoted and single-quoted strings (a & inside single quotes is not a macro reference), a block-comment state and a statement-comment state

Vocabulary

The word lists are resolved from the SAS language data published with the SAS extension for Visual Studio Code (sassoftware/vscode-sas-extension), so the grammar colours the words SAS actually documents rather than a hand-written sample. 2096 words in 13 groups:

group	count	group	count
statements	96	procNames	386
functions	640	callRoutines	105
macroStatements	30	macroFunctions	65
options (data set)	76	systemOptions	344
odsTagsets	34	styleElements	172
styleAttributes	101	styleLocations	6
sqlKeywords	41

keywords.js regenerates them, matching the convention in sql and mysql:

node keywords.js /path/to/vscode-sas-extension/server/data

Tests

sas.test.ts follows the existing testTokenization style, with 8 stateful groups covering statement position, macro statements, comments and strings, PROC SQL, ODS and OPTIONS, and instream data.

npm test        # check-samples plus all tokenization tests: 87 passed

Notes for review

• CONTRIBUTING.md describes the older layout (src/basic-languages/{lang}/ and monaco.contribution.ts). The current layout is src/languages/definitions/{lang}/ with the list in definitions/register.all.ts, which is what this change follows.
• build/check-samples requires a sample file per language directory, hence website/src/website/data/home-samples/sample.sas.txt.
• Per-procedure options such as noprint on PROC MEANS are not in the flat vocabulary. Recognising them needs a state per procedure, so they are left out of this change rather than approximated.
• npx tsc -p ./src reports no new errors. The two it does report (src/index.ts cannot resolve @vscode/monaco-lsp-client, and a regex flag in wgsl.ts under the es5 target) are present on main before this change.

Checklist

• grammar, language configuration and registration
• tests, following the existing tokenization test style
• sample file for the website
• npm test passes
• no new type errors

[Allan Bowe (allanbowe)]

@microsoft-github-policy-service agree