Skip to content

Fix Pandoc subdocument XML namespace handling - #658

Open
aka-debug-jie wants to merge 2 commits into
elapouya:masterfrom
aka-debug-jie:fix/issue-620-pandoc-namespaces
Open

Fix Pandoc subdocument XML namespace handling#658
aka-debug-jie wants to merge 2 commits into
elapouya:masterfrom
aka-debug-jie:fix/issue-620-pandoc-namespaces

Conversation

@aka-debug-jie

@aka-debug-jie aka-debug-jie commented Aug 3, 2026

Copy link
Copy Markdown

Why

Pandoc-generated DOCX files can declare DrawingML namespaces on the document
root rather than on each body child. Removing the serialized w:body wrapper
then leaves unbound prefixes in the inserted fragment, so Word-compatible
parsers cannot reopen the rendered document.

Root cause

The original implementation serialized the whole body and removed the wrapper,
including declarations required by descendants. The first fix serialized every
child independently, which preserved correctness but repeated inherited
namespace declarations on every child and caused severe fragment growth.

Change

  • create one temporary wrapper with the destination body tag and namespace map
  • deep-copy all source body children into that wrapper and serialize once
  • remove only the temporary outer wrapper while preserving missing or
    conflicting declarations on inserted content
  • keep the existing direct sectPr removal behavior
  • make the tracked CI launcher discover tests relative to itself, use
    sys.executable, propagate child failures, and emit a deterministic
    SHA-256 RUN/PASS ledger

Validation

  • reproduced the original unbound-prefix failure on the upstream baseline
  • proved the reviewed per-child implementation fails the final 10,000-paragraph
    regression with an 11,800,000-byte fragment
  • verified the final fragment retains all 10,000 paragraphs below the
    1,000,000-byte structural bound
  • reopened the rendered DOCX with python-docx and verified the image
    relationship, DrawingML/PictureML nodes, namespace conflicts, and final
    sectPr placement
  • passed the repository-native suite, strict per-script suite, related
    subdocument/header-footer checks, flake8, wheel/sdist builds, and archive
    audits in both review phases
  • opened, saved, and reopened the generated document in Windows Word 2024
    without a repair prompt and confirmed the image renders

Scope and risks

This changes body-fragment serialization and the tracked test launcher only.
It does not change public APIs, template syntax, dependencies, relationships,
fixtures, headers, footers, or footnotes. The launcher change ensures this PR's
regression is a real CI-discovered test; broader nested-command cleanup remains
separate from this issue. Remote GitHub Actions still require repository-side
approval to run.

Closes #620

@yangfan-yf-yf yangfan-yf-yf left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for two blocking issues:

  1. Subdoc._get_xml() now serializes each direct body child independently. lxml writes every inherited namespace declaration onto every child, including namespaces that are not used by that child or are already available from the destination document. In a CPython 3.12.3 / lxml 6.1.1 benchmark with a 10,000-paragraph dynamic subdocument, the intermediate fragment grew from 340,000 bytes at 1f143fb to 11,800,000 bytes at 62e0075; median render() time across three fresh runs grew from 0.083 s to 2.400 s. The same path affects every subdocument, not only Pandoc input. Please preserve only the missing or conflicting declarations needed by the inserted content and add a scale regression.

  2. tests/pandoc_subdoc.py is not exercised by the current test job. The job invokes python tests/runtests.py from the repository root, but the runner globs [A-Za-z]*.py relative to the current directory, so it runs setup.py rather than the scripts under tests. It also ignores each subprocess.call() return code. Against 1f143fb, the new regression script exits 1 with the reported namespace error, while runtests.py still exits 0. Please discover tests relative to file and propagate child-process failures, or invoke this regression directly from the test job.

The namespace fix itself handles the reported document correctly: the Pandoc 3.9 reproduction fails at the base with the unbound a prefix and succeeds at this head with the image relationship, a/pic elements, nested section properties, and one final body sectPr intact. Custom default namespaces, prefixed attributes, and conflicting prefix mappings also remained resolved.

@aka-debug-jie

Copy link
Copy Markdown
Author

Thanks for the detailed reproduction. Both requested changes are addressed in
the follow-up commit.

  1. _get_xml() now deep-copies the source body children under one temporary
    wrapper carrying the destination body's namespace context and serializes the
    wrapper once. The final regression keeps exactly 10,000 paragraphs while
    enforcing a fragment smaller than 1,000,000 bytes. The reviewed per-child
    implementation fails that same test at 11,800,000 bytes; the final code
    passes it.
  2. tests/runtests.py now resolves the tests directory from __file__, runs
    children with sys.executable and check=True, and emits a sorted SHA-256
    RUN/PASS ledger. Running the launcher from the repository root includes
    tests/pandoc_subdoc.py, and both the repository-native and strict suites
    pass.

The follow-up also passes the related subdocument/header-footer checks,
flake8, wheel/sdist builds, archive audits, and a Windows Word open/save/reopen
check. The runner change here is limited to making this PR's regression a real
CI-discovered gate; it does not claim the separate module_execute.py cleanup
proposed in #652.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Subdocument merge with Pandoc-generated DOCX causes XML namespace errors when opening the resulting file

2 participants