Skip to content

Add Video OCR: extract burned-in (hardcoded) subtitles#12270

Merged
niksedk merged 1 commit into
mainfrom
feature/video-ocr-burned-in
Jul 8, 2026
Merged

Add Video OCR: extract burned-in (hardcoded) subtitles#12270
niksedk merged 1 commit into
mainfrom
feature/video-ocr-burned-in

Conversation

@niksedk

@niksedk niksedk commented Jul 8, 2026

Copy link
Copy Markdown
Member

Implements video OCR of hardcoded/burned-in subtitles (#10944), inspired by VideOCR's pipeline.

New feature: Video → OCR burned-in subtitle...

![screenshot placeholder — drag & drop a screenshot here]

How it works

  1. Scan area selection — a video frame preview with a draggable/resizable crop rectangle (custom CropAreaSelector control: move, edge/corner resize, draw-new, dimmed outside area, rule-of-thumb presets for bottom third / bottom half / full frame). A slider scrubs the preview through the video. Default: bottom third.
  2. Frame extraction — ffmpeg samples the scan area at a configurable rate (default 5 fps) into near-lossless JPEGs (fps + crop + optional scale down to 720 px).
  3. Frame de-duplication before OCR — frames are compared on small brightness-thresholded masks using Jaccard overlap, which follows the (bright) subtitle text rather than the moving video behind it. Consecutive near-identical frames form one group; only the middle frame of each group is OCR'ed, and frames with no bright pixels are skipped entirely as blank. This is what makes whole-movie OCR practical.
  4. OCR engines
    • Paddle OCR (standalone on Windows/Linux with the existing auto-download flow, or Python) — reuses the existing PaddleOcr batch engine via a new decode-free file-input path (PaddleOcrBatchInput.SourceFileName), so the whole video is one batch: no per-image decode/re-encode, flat memory, one model load.
    • Ollama vision — e.g. the local glm-ocr model (reuses the existing OllamaOcr engine).
    • GLM API (new engine) — GLM vision models via the Z.ai / bigmodel.cn OpenAI-compatible chat completions API with API key, model and endpoint settings.
  5. Line building — consecutive OCR results are merged by whitespace/case-insensitive Levenshtein similarity (default ≥ 80%) with the on-screen-longest text winning (majority vote), gaps ≤ 250 ms bridged (so a one-frame brightness flicker doesn't split a line), and blips shorter than 250 ms dropped. Vision-LLM hallucinations on textless frames (markdown fences, prompt echoes, symbol-only lines) are sanitized to empty, which correctly means "no subtitle".
  6. ASSA position — optional {\anX} tag derived from the scan area position (e.g. {\an8} for a top scan area; bottom-center emits no tag).

Lines appear live in the results grid during OCR; OK loads them into the main window (with ASSA formatting when the current format is Advanced Sub Station Alpha).

Also in this PR

  • PaddleOcrInstallHelper — the Paddle engine/models download prompts extracted to a shared helper (previously duplicated across view models), used by the new feature.
  • Wired into the Avalonia menu, the native macOS menu bar, the shortcut system (assignable in Options → Shortcuts), DI, settings (SeVideoOcr, crop stored as percentages so it applies across resolutions), localization (LanguageVideoOcr), and docs (docs/features/video-ocr.md, F1 help).
  • 32 unit tests for the merge/majority-vote/similarity/cleaning/alignment logic.

Verification

  • Full solution builds with 0 warnings; all 604 UI tests pass.
  • End-to-end test on a synthetic video (three subtitles burned over a moving testsrc2 background) through the real pipeline with Ollama + glm-ocr: output was exactly 3 lines with exact texts and frame-accurate timings (1–4 s, 5–8 s, 10–14 s).
  • An 8-angle self-review pass was run on the diff; all 10 findings (including macOS menu wiring, GLM JSON escaping, flicker line-splitting, engine-error reporting, preview bitmap leak) were fixed and re-verified.

Closes #10944

🤖 Generated with Claude Code

New "Video -> OCR burned-in subtitle..." feature (#10944), inspired by
timminator's VideOCR pipeline:

- ffmpeg samples frames from a user-selected scan area (draggable/resizable
  crop rectangle over a video frame preview, with bottom-third default and
  presets) at a configurable rate
- consecutive near-identical frames are collapsed before OCR by comparing
  bright-pixel masks (Jaccard overlap), so each on-screen subtitle is
  OCR'ed once; frames with no bright pixels are skipped as blank
- OCR engines: Paddle OCR (standalone/Python, reusing the existing batch
  engine via a new decode-free file input path), Ollama vision (e.g.
  glm-ocr), and a new GLM API engine (Z.ai / bigmodel.cn, OpenAI-compatible)
- consecutive OCR results are merged into subtitle lines by Levenshtein
  text similarity with majority-vote text selection, max-gap bridging and
  minimum-duration filtering; vision-LLM hallucinations on textless frames
  are sanitized away
- optional ASSA position tag ({\anX}) derived from the scan area position
- shared Paddle install prompts extracted to PaddleOcrInstallHelper

Verified end-to-end against a synthetic video with burned-in subtitles
over a moving background (Ollama + glm-ocr): exact texts and timings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@niksedk niksedk merged commit e66a672 into main Jul 8, 2026
3 checks passed
@niksedk niksedk deleted the feature/video-ocr-burned-in branch July 8, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Video OCR

1 participant