Fix speech to text add periods for non Latin punctuation#12248
Merged
niksedk merged 1 commit intoJul 7, 2026
Conversation
The add periods step of speech to text post processing only recognized Latin line endings (period, question mark, comma, and so on), so a line ending with an Arabic question mark or Arabic comma was treated as unterminated and got a Latin period appended, producing lines ending in "?." or ",." in Arabic transcriptions. The same applied to Urdu, ellipsis, and CJK terminators. The ending check now uses one shared list of line terminating characters that includes the Arabic question mark, comma, and semicolon, the Urdu full stop, the ellipsis, fullwidth CJK punctuation, and closing quotes, in both the paragraph loop and the final paragraph check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The "add periods" step of speech to text post processing produced lines ending in "؟." and "،." in Arabic transcriptions: a Latin period appended right after an Arabic question mark or comma.
The step appends a period to a line followed by a speech gap unless the line already ends with punctuation, but the ending check only recognized Latin characters (period, exclamation mark, question mark, comma, colon, closing brackets). The Arabic question mark, Arabic comma, and similar non Latin terminators were treated as unterminated text. The same applies to Urdu, ellipsis, and fullwidth CJK punctuation.
This also explains why the doubling looked random: only lines followed by a long enough pause were affected, so back to back lines stayed clean while lines before a scene change got the extra period.
Fix
Both ending checks (the paragraph loop and the final paragraph) now use one shared list of line terminating characters that additionally includes the Arabic question mark (U+061F), Arabic comma (U+060C), Arabic semicolon (U+061B), the Urdu full stop (U+06D4), the ellipsis, fullwidth CJK terminators, and closing quotes.
Verified on a 45 minute Arabic transcription where dozens of lines previously came out as "...؟." with post processing enabled; with the fix the same lines end in "؟" alone, and Latin language behavior is unchanged.