Skip to content

Python: make the memory markdown escape invertible for backslash runs - #9728

Closed
Manohar Paturi (ManoharPaturi) wants to merge 2 commits into
microsoft:mainfrom
ManoharPaturi:fix/memory-escape-roundtrip
Closed

Manohar Paturi (ManoharPaturi) wants to merge 2 commits into
microsoft:mainfrom
ManoharPaturi:fix/memory-escape-roundtrip

Conversation

@ManoharPaturi

@ManoharPaturi Manohar Paturi (ManoharPaturi) commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Motivation & Context

_escape_markdown_line prepended one backslash to any line starting with # or \#, but _unescape_markdown_line only stripped a backslash from a body starting with exactly \#. A line that already began with an escaped hash therefore gained a backslash on every write and read cycle, and the corruption compounded across the rereads that auto loading and consolidation perform. Reproduced on main with a one line round trip: \# literal fact came back as \\# literal fact.

Description & Review Guide

  • What are the major changes? The escape maps a run of n backslashes before a hash to 2n + 1, and the reader reduces an odd run back to the original n. A plain # line still gets the single backslash it always had, which is the n = 0 case.
  • What is the impact of these changes? Every backslash run length round trips exactly. Even runs are not produced by this writer, so the reader passes them through untouched; that also keeps lines written by the previous scheme from being mangled further.
  • What do you want reviewers to focus on? The round trip test covers a plain hash line, a single escaped hash, and a two backslash run, and fails on main.

Related Issue

Fixes #9727

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi) One material backward-compatibility regression remains in python/packages/core/agent_framework/_harness/_memory.py:145-148. The new decoder assumes every odd backslash run was produced by the new encoding, but the previous writer left runs of two or more backslashes unchanged. Existing unversioned files with three, five, or seven backslashes before # therefore lose characters when loaded and persist the corruption on rewrite. Please make the new encoding distinguishable, retain legacy decoding for unversioned files, and add compatibility tests that read odd runs written by the previous implementation.

A line starting with backslash hash used to gain one more backslash on write that the reader never stripped, so every reload corrupted the memory a little more. Runs of n originals are now written as 2n+1 backslashes, which decodes exactly, and every written record carries a format marker identifying the invertible encoding. Files without the marker were written by the previous implementation, which left runs of two or more backslashes verbatim, so they are decoded with the previous reader's exact behavior: strip one backslash only from a body starting backslash hash, and leave every longer run untouched. Compatibility tests load odd runs written by the previous implementation from unversioned files and assert they survive unchanged.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Fixed in 72698c1, with your compatibility case reproduced first. The new encoding is now distinguishable: every written record carries a format marker, and on read the marker selects the decoder.

Files without the marker were written by the previous implementation, which left runs of two or more backslashes verbatim, so they are decoded with the previous reader's exact behavior: strip one backslash only from a body starting backslash hash, and leave every longer run untouched. Compatibility tests load unversioned files holding three, five, and seven backslashes before a hash and assert they survive unchanged rather than losing characters, plus an unversioned single-backslash line still decoding exactly as before. Versioned files round trip every run length from zero through five exactly.

The unversioned odd-run test fails on the previous head and passes now, the memory suite passes at 38 tests, and ruff check and format are clean.

@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Eduard van Valkenburg (@eavanvalkenburg) the PR body now carries the required Contribution Checklist with the non-breaking item confirmed, matching the template. Everything from your review is addressed in the latest push and the thread replies have the verification details.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Manohar Paturi (@ManoharPaturi) The requested unversioned backslash cases now work, but format detection introduces a separate content-boundary regression in python/packages/core/agent_framework/_harness/_memory.py:539-545.

A record with summary <!-- agent-framework-memory-escape: v2 --> and memory Keep the deployment instructions. does not round-trip: the summary is discarded and reloads as Keep the deployment instructions. instead. The parser consumes marker-shaped summary content as metadata. The substring test at line 539 also lets an ordinary legacy bullet mentioning that marker select v2 decoding for the entire unversioned file.

Please recognize and consume a standalone format marker only in the metadata header, before content sections, and preserve the same text inside summaries or memories. Add a summary round-trip regression and a legacy-body marker case.

The format marker is recognized as a standalone line before any content
section, so marker shaped summary or memory text stays content and a
legacy body line mentioning the marker no longer switches decoding.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Fixed in c4a3e00, with both shapes reproduced on the previous head first.

The marker is now recognized and consumed only as a standalone line in the metadata header, before any content section, and the substring pre-scan is gone. A summary holding exactly the marker text round trips as summary content next to its memory instead of being swallowed and backfilled from Keep the deployment instructions., and the same text inside a memory stays a memory.

For the legacy side, a body line that merely mentions the marker no longer selects v2 decoding for the whole file. The regression stores such a mention next to a three backslash memory and asserts both survive verbatim under legacy decoding, matching the same file without the mention. The escape feature itself is untouched: the run length sweep from zero through five still round trips exactly.

Both regressions fail on the previous head and pass now. The memory suite passes at 40 tests and ruff check and format are clean.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Manohar Paturi (@ManoharPaturi), thanks for taking the time to investigate and implement this. We want to review and agree on #9727 before accepting an implementation PR, rather than evaluating the issue and implementation together. I’m closing this PR for now. Please wait for maintainer direction on the issue before opening a new implementation PR.

This branch was previously deployed

1 inactive deployment
github-app-auth — c4a3e000 Deployed Oct 29, 2026 by ManoharPaturi via add_label #26493
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: memory lines starting with backslash hash gain a backslash on every round trip

2 participants