Syntax highlighting using Prism.parse_lex

[tompng]

Completely drops ripper dependency.

Drops ruby-node css class used to colorize dstring and backref

backref($&) is now colored as identifier, same as global variable.
dstring("#{expr}", `#{expr}`) is now colored as string. Embedded expression parts are colored separately.

Stops token squashing:

"string" was a single string token. Changed to " + string + "
"string#{:symbol}" was a single dstring token. Changed to string tokens " string #{ } ". : and `symbol is colorized as symbol color.
So there should be lots of diff.

Before

After

Fix some unkown bugs

# Documenting this code crashed
def foo
  <<~A; :\
  A
  b
end

[Copilot]

Pull request overview

This pull request replaces the Ruby syntax-highlighting/tokenization pipeline with a Prism-based implementation (Prism.parse_lex), removing the prior Ripper-based lexer and adjusting HTML/CSS/test expectations accordingly. This aligns RDoc’s Ruby parsing/highlighting with Prism’s token stream while improving handling of interpolated literals and previously crashing edge cases.

Changes:

• Switch Ruby parsing/token collection to Prism.parse_lex and introduce RDoc::Parser::RubyColorizer for producing a colored token stream.
• Update HTML generation + CSS to reflect new token kinds and finer-grained tokenization (no more ruby-node).
• Refresh and add tests covering partial colorization, interpolation, heredocs, multibyte content, and method token streams.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
lib/rdoc/parser/ruby.rb	Uses Prism.parse_lex tokens and delegates node-range highlighting to RubyColorizer.
lib/rdoc/parser/ruby_colorizer.rb	New Prism-based colorizer producing ColoredToken streams for full/partial ranges.
lib/rdoc/token_stream.rb	Updates token kind → CSS class mapping to match new colorizer output.
lib/rdoc/markup/to_html.rb	Uses RubyColorizer for Ruby verbatim highlighting instead of the removed Ripper lexer.
lib/rdoc/generator/markup.rb	Removes Ruby-specific indentation prepend that relied on :char_no from the old lexer.
lib/rdoc/parser/ripper_state_lex.rb	Removed (drops Ripper dependency for tokenization).
lib/rdoc/generator/template/darkfish/css/rdoc.css	Removes .ruby-node styling.
lib/rdoc/generator/template/aliki/css/rdoc.css	Removes .ruby-node styling (including dark theme selector).
test/rdoc/rdoc_token_stream_test.rb	Updates token kinds and expected HTML spans to match new mapping.
test/rdoc/parser/ruby_test.rb	Adds coverage for code-object token stream text output.
test/rdoc/parser/ruby_colorizer_test.rb	New tests validating Prism-based colorization across multiple Ruby constructs.
test/rdoc/markup/to_html_test.rb	Updates expectations to match new token splitting/interpolation handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kou]

+1

[matzbot]

🚀 Preview deployment available at: https://bd4fca4d.rdoc-6cd.pages.dev (commit: 78ea670)