All pull requests

Pull requests

• ci(python): run the tests against a release build

  #2593·agarg21 opened Oct 26, 2026·

  ·

• fix(bpe): decorate bytes, not characters, in byte-level models with affixes

  #2592·patrasq opened Oct 26, 2026·

  ·

• fix: canonicalize singleton Metaspace sequences

  #2591·original4422 opened Oct 26, 2026·

  ·
  1

• fix(wordlevel): don't give special tokens a second id when they appear in the corpus

  #2588·ViderStar opened Oct 25, 2026·

  ·

• chore(deps): bump the actions group with 5 updates

  dependenciesgithub_actions
  #2587·dependabot[bot] opened Oct 25, 2026Bot·

  ·
  1

• Fix byte-level decoding with configured decoder chains

  #2586·AryanRG opened Oct 25, 2026·

  ·

• Restore stride and overflowing encodings in v1

  #2585·pma1999 opened Oct 24, 2026·

  ·
  1

• fix(unigram): preserve byte fallback during training

  #2584·original4422 opened Oct 24, 2026·

  ·
  1

• fix(tk-encode): update ptr_hash to 2.1.3, fix forked Tokenizer build

  #2581·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

  2

• feat(composition): component handles

  #2579·SBrandeis opened Oct 22, 2026Collaborator·2/4·

  ·
  1

• feat(composition): ComponentSequence

  #2577·SBrandeis opened Oct 22, 2026Collaborator·4/4·

  ·
  1

• feat(composition): post-processors

  #2573·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): decoders

  #2572·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): pre-tokenizers

  #2571·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): normalizers

  #2570·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• feat(composition): models and Tokenizer(model)

  #2569·SBrandeis opened Oct 22, 2026Collaborator·

  ·
  1

• Respect recognized cl100k numeric group sizes

  #2568·Aurantium-47 opened Oct 22, 2026·

  ·

• Fix Kimi pretokenization for overlapping Han categories

  #2567·Aurantium-47 opened Oct 22, 2026·

  ·
  1

• feat(composition): serialization utils

  #2563·SBrandeis opened Oct 21, 2026Collaborator·3/4·

  ·
  1

• feat(composition): TokenizerSpec: JSON config as source of truth

  #2562·SBrandeis opened Oct 21, 2026Collaborator·1/4·

  ·
  7

• fix(bitcannon): backwards compatibility: use onig to generate atom_tables

  #2558·SBrandeis opened Oct 21, 2026Collaborator·

  ·
  1

  1

• Restore Python tokenizer JSON serialization for v1

  #2556·xgbtxy opened Oct 20, 2026·

  ·
  1

  1

• fix(vocab): load an empty vocab

  #2554·SBrandeis opened Oct 19, 2026Collaborator·

  ·

  · v1.0.0
  3

• feat(python): Add support for DecodeStream in Python

  #2551·SBrandeis opened Oct 19, 2026Collaborator·

  ·
  1

  1

• fix(byte-level): drop unknown bytes + warn

  #2550·SBrandeis opened Oct 19, 2026Collaborator·

  ·
  1

  1