Skip to content

fix(html): remove broken obsolete heading hook - #2618

Open
Thota Shashank (thotashashank302) wants to merge 3 commits into
microsoft:mainfrom
thotashashank302:fix/238-obsolete-heading-override
Open

Thota Shashank (thotashashank302) wants to merge 3 commits into
microsoft:mainfrom
thotashashank302:fix/238-obsolete-heading-override

Conversation

@thotashashank302

@thotashashank302 Thota Shashank (thotashashank302) commented Oct 5, 2026 •

Copy link
Copy Markdown

Closes #238.

A literal <hn> or <HN> element accidentally dispatches _CustomMarkdownify.convert_hn, whose signature is incompatible with markdownify's element callbacks. HTML conversion then fails, and MarkItDown falls back to plain text, returning the original HTML instead of Markdown:

<p>Before</p><hn>Literal hn</hn><p>After</p>

With this fix, the output is:

Before

Literal hn

After

Remove the obsolete hook. Normal h1–h6 rendering already uses markdownify's inherited heading renderer and supplies the needed separators. Require markdownify>=1.0.0: the old convert_hn hook was still active in 0.13.x, so removing it without a dependency bound could change normal heading spacing in older installations.

Add two end-to-end regression cases for <hn> and <HN> (both fail against the base implementation) and ten behavior-preservation cases for all six heading levels beside bare text, adjacent headings with ATX/ATX_CLOSED/UNDERLINED styles, and inline emphasis.

Validation on macOS ARM64 against upstream 7482c09 (October 26, 2026):

  • Python 3.12 / markdownify 1.2.3: core suite 1,130 passed, 57 skipped (GITHUB_ACTIONS=true python -m pytest packages/markitdown/tests -q -ra). Skips cover external services, native Windows paths, and optional ExifTool.
  • Python 3.14 via hatch test --python 3.14 -- -q -ra: 1,130 passed, 57 skipped, with one existing nturl2path deprecation warning.
  • Minimum markdownify 1.0.0: 46 HTML tests passed.
  • pre-commit run --all-files, wheel/sdist builds, wheel dependency metadata, CLI smoke test, and git diff --check: passed.

@thotashashank302

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@kokokoXUY XU (kokokoXUY) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: the removal is correct for markdownify >= 1.0 and I would merge it. Two things are worth addressing first: the new tests also pass on the base commit, so they do not cover the only behaviour that changes, and the markdownify dependency is unpinned, so "obsolete" depends on the version that gets resolved.

What I checked locally (base 4cc9fa1 -> head 0924b60, markdownify 1.2.3 as resolved here):

  • Dispatch. markdownify 1.2.3 builds the hook name from the tag (convert_%s, markdownify/init.py:385-397) and only falls back to the heading regex at :399-403, which calls convert_hN(n, el, text, parent_tags). That release has no convert_hn, so _CustomMarkdownify.convert_hn cannot run for h1-h6. The PR description is right that the override is dead code there.
  • It is not unreachable in every case, though. A literal <hn> tag produces exactly the name convert_hn, and the dispatcher calls it as convert_fn(node, text, parent_tags=parent_tags) (markdownify/init.py:324). On base that raises TypeError: _CustomMarkdownify.convert_hn() missing 1 required positional argument: 'text'; the exception is collected by _markitdown.py:669-676 and the generic plain-text converter silently takes over. End to end on base, MarkItDown().convert_stream(b"<p>Before</p><hn>Literal hn</hn><p>After</p>") returns the raw HTML '<p>Before</p><hn>Literal hn</hn><p>After</p>' and no error reaches the caller; on head the same input returns 'Before\n\nLiteral hn\n\nAfter'. So this is a small bug fix as well as a cleanup.
  • Corpus. I converted 33 cases through HtmlConverter (h1-h6, heading with text before/after, headings in lists, tables, divs and 40-deep nesting, ATX / ATX_CLOSED / UNDERLINED, inline markup, and the repo's test_blog.html, test_serp.html, test_wikipedia.html). Base and head are byte-identical except the three <hn> / <HN> cases.
  • Tests. packages/markitdown/tests/test_html.py is 34 passed on base and 44 passed on head (the 10 new cases). I also ran the head file against the base source: 44 passed, so the new cases are behaviour-preservation coverage, not regression coverage. A <hn> case is red on base and green on head (2 failed / 2 passed).
  • Version range. packages/markitdown/pyproject.toml:30 is the bare string "markdownify" and the repo has no lockfile, so an environment can still be on a release where this hook was live. Forcing markdownify 0.13.1 into the same corpus changes 14 of 33 cases: 0.13.x still exposes convert_hn (1.0.0 and 1.1.0 renamed it _convert_hn, 1.2.x uses convert_hN), so base keeps the separator the override added and head does not, e.g. base 'Before\n# Heading\n\nAfter' vs head 'Before# Heading\n\nAfter', and the three real-world vectors change length too (blog 10903 -> 10898, serp 70208 -> 70191, wikipedia 192118 -> 192086).

What I would like to see before this lands:

  1. One regression case for the tag that actually dispatches convert_hn (<hn>, and the uppercase variant). Without it, nothing in the suite fails if that name comes back.
  2. A sentence in the description, or a lower bound in pyproject, making the supported markdownify range explicit, since the cleanup assumes >= 1.0. If older markdownify is not supported, a bound also prevents this class of divergence.

Not in scope, same file and same era mismatch: convert_img still takes convert_as_inline, which the 1.x dispatcher never passes (it sends parent_tags= as a keyword), so that branch is dead. It is latent because upstream only collapses inline images when keep_inline_images_in is set; I confirmed the value received is always False on 1.2.3.

@thotashashank302 Thota Shashank (thotashashank302) changed the title fix(html): remove obsolete heading conversion override fix(html): remove broken obsolete heading hook Oct 6, 2026
@thotashashank302

Copy link
Copy Markdown
Author

Thanks for the thorough review and for catching the <hn> fallback behavior!

I've addressed both points in commit 200660a:

  • Added regression tests for <hn> and <HN> tags (which fail against the base implementation).
  • Added the markdownify>=1.0.0 lower bound to pyproject.toml and updated the PR description accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

convert_hn isn't being used properly

2 participants