Repository navigation
Conversation
The PDF parser assumed the legacy self-hosted contract: a single
POST /file_parse answering inline with md_content. The current
self-hosted /tasks API and the online batch API are task-based
(create -> poll -> zip download), so both made OpenViking fail:
the online service has no /file_parse endpoint at all (404) and no
/health endpoint for the startup preflight.
- add pdf.mineru_api_mode ("auto" | "sync" | "async", default "auto"):
auto detects the flavor from the first response, so existing v1
deployments keep working without config changes
- add pdf.mineru_token, sent as Authorization: Bearer on every call
- v2-tasks flow: POST /tasks -> poll status_url (pending/processing/
completed) -> download the result zip
- online-batch flow: POST /extract/task/batch -> poll
extract-results/batch/{batch_id} until state=done -> download
full_zip_url
- unpack the zip via safe_extract_zip, persist images/ into the media
store and rewrite markdown references (mirrors the v1 image path)
- skip the /health startup preflight for the async flavors (the
online service has no such endpoint)
- docs (en/zh): protocol table and online example
Fixes volcengine#4708
Enhancement: Add mineru-first strategyMotivation当前 �uto 策略是 本地 pdfplumber 优先 → MinerU 回退。但对于需要高精度解析的场景(复杂版面、表格、公式),用户可能希望 MinerU 优先 → 本地 pdfplumber 回退。 在本地测试中,MinerU 的解析质量明显优于 pdfplumber,尤其是表格结构识别、数学公式保留、多栏版面处理。因此建议增加 mineru-first 策略。 Changes1. parser_config.py - 验证逻辑`python Before:if self.strategy not in ("local", "mineru", "auto"): After:if self.strategy not in ("local", "mineru", "auto", "mineru-first"): 2. pdf.py - 新增策略分支
3. core.py - 预检逻辑`python Before:pdf_config.strategy == "mineru" After:pdf_config.strategy in ("mineru", "mineru-first") Strategy Comparison
Test Results已验证 mineru-first 策略正常工作:
|
Fixes #4708
Problem
The PDF parser only implemented the legacy self-hosted MinerU contract: one
POST {mineru_endpoint}/file_parseanswering inline withmd_content. Both newer deployments break today:POST /tasks→202→status_url/result_url→ zip result) — the inline endpoint is gone;mineru.net/api/v4) has no/file_parse(404), no/health(so the startup preflight hard-fails the service), and requires a Bearer token that had nowhere to be configured.Change
pdf.mineru_api_mode("auto"|"sync"|"async", default"auto"):autodetects the protocol from the first response — inlinecompleted+resultsstays v1; a task-shaped payload (or a 404 on/file_parse) switches to the task flow — so existing v1 deployments need zero config changes.pdf.mineru_tokensent asAuthorization: Bearer ...on every call.POST /tasks→ pollstatus_urluntilstatus=completed→ downloadresult_urlzip.POST /extract/task/batch→ pollextract-results/batch/{batch_id}untilstate=done→ downloadfull_zip_url.safe_extract_zip(zip-slip guarded), persistsimages/into the media store and rewrites markdown references — mirroring the existing v1 image path./health) is skipped for the async flavors (the online service has no such endpoint; this was part of the reported startup errors).Wire-format evidence: the v2 contract was taken from the official
mineru/cli/api_client.py(submit → 202 →status_url/result_url,pending/processing/completed); the online endpoints were verified live (/api/v4/extract/task/batchand/api/v4/extract-results/batch/{id}respond401 login requiredwithout a token,/api/v4/file_parseresponds404— which is exactly the reported failure).Testing
tests/parse/test_pdf_mineru.py(new, 14 tests): v1 unchanged undersyncandauto; v2 flow end-to-end (202 → poll → zip → markdown + image rewrite); online batch flow (with 404 fallthrough from/tasks); auto fallback on 404; task failure detail propagation; polling timeout; missing-task-endpoint actionable error; zip without markdown; config validation.All green locally (48 passed incl.
test_mineru_preflight,test_pdf_bookmark_extraction,test_parser_config_wiring,test_markdown_local_image_refs).