mojibake and character corruption improvements

AGENTS.md

@@ -3,6 +3,7 @@ A skill is a set of local instructions to follow that is stored in a `SKILL.md`
 
 ### Available skills
 - new-api-guide: Guide users through the APIOps Cycles method to design and scope a new API from business need to publishing and improvement. Use when the user needs help deciding the current station, checking whether entry or exit criteria are met, selecting the right next resource or canvas from the canonical method data, gathering answers for the selected canvas, or moving from method guidance into canvas JSON authoring and export without jumping too early into API design. (file: skills/new-api-guide/SKILL.md)
+- localized-text-editing: Safely edit APIOps method and canvas labels containing localized or accented text without introducing mojibake, replacement characters, or Windows shell encoding damage. Use when changing `src/data/**` localized JSON, station labels, resource labels, canvas text, or any non-ASCII method content. (file: skills/localized-text-editing/SKILL.md)
 - canvas-import-json-authoring: Create or review importable CanvasCreator JSON files using schema-first validation, template-specific section coverage, locale-aware section descriptions, and APIOps metadata defaults. Use when generating new filled canvas examples, converting raw notes into import JSON, or validating existing import files before import or export. (file: node_modules/canvascreator/.agents/skills/canvas-import-json-authoring/SKILL.md)
 - common-contributor-workflow: Follow the standard CanvasCreator contribution flow for scoping, implementation, validation, docs hygiene, and pull request preparation. Use when working on CanvasCreator package changes or package-bundled skill maintenance. (file: node_modules/canvascreator/.agents/skills/common-contributor-workflow/SKILL.md)
 - export-cli-usage-patterns: Create, review, or troubleshoot CanvasCreator export CLI usage for `scripts/export.js` and `canvascreator-export`, including argument patterns, output-file expectations, and format-specific behavior for `json`, `svg`, `pdf`, and `png` exports. (file: node_modules/canvascreator/.agents/skills/export-cli-usage-patterns/SKILL.md)
@@ -21,3 +22,11 @@ A skill is a set of local instructions to follow that is stored in a `SKILL.md`
   2. Use `canvas-import-json-authoring` to create or validate importable canvas JSON.
   3. Use `export-cli-usage-patterns` to export or troubleshoot exported artifacts.
 - Context hygiene: Keep context small and avoid loading unnecessary reference material.
+
+## Localized text and encoding safety
+
+- Treat all JSON files under `src/data/` as UTF-8.
+- Do not rewrite localized or accented text through PowerShell heredocs or ad hoc shell text replacement. On Windows this can silently turn characters such as `ã`, `ç`, `é`, and smart punctuation into mojibake or `?`.
+- Prefer `apply_patch` for small exact edits. For scripted edits, use Node with `readFileSync(file, "utf8")` and preserve existing text; when inserting non-ASCII text from a script, use Unicode escapes or another byte-safe source.
+- Run `npm.cmd test` on Windows. Plain `npm test` can fail through PowerShell execution policy because it invokes `npm.ps1`.
+- The content integrity test fails on known mojibake and replacement patterns including `Ã`, `Â`, `â...`, `�`, `Snum`, `snum`, and `cumprems`.

scripts/test-method-content-integrity.mjs

@@ -29,6 +29,104 @@ const findings = [];
 const warnings = [];
 const lifecycleStages = new Set(["strategy", "architecture", "design", "delivery", "publishing", "improving"]);
 
+const textQualityRoots = [
+  path.join("src", "data", "method"),
+  path.join("src", "data", "canvas")
+];
+const textQualityPatterns = [
+  {
+    name: "Unicode replacement character",
+    pattern: /\uFFFD/u,
+    reason: "usually means invalid bytes were decoded with replacement"
+  },
+  {
+    name: "UTF-8 mojibake lead U+00C3",
+    pattern: /\u00c3/u,
+    reason: "usually means UTF-8 text was decoded as Windows-1252 or Latin-1"
+  },
+  {
+    name: "UTF-8 mojibake lead U+00C2",
+    pattern: /\u00c2/u,
+    reason: "usually means a non-breaking space or punctuation was double-decoded"
+  },
+  {
+    name: "UTF-8 punctuation mojibake",
+    pattern: /\u00e2[\u0080-\u009f\u20ac\u201a\u201c\u201d\u201e\u02dc\u2019\u2018\u2013\u2014]/u,
+    reason: "usually means smart quotes, dashes, or zero-width characters were corrupted"
+  },
+  {
+    name: "question-mark accent replacement",
+    pattern: /\b[A-Za-z\u00c0-\u017e]*\?[A-Za-z\u00c0-\u017e]*\b/u,
+    reason: "usually means an accented character was replaced by ? during shell rewriting"
+  },
+  {
+    name: "known Portuguese corruption Snum",
+    pattern: /\bSnum\b/u,
+    reason: "expected text is likely Sem"
+  },
+  {
+    name: "known Portuguese corruption snum",
+    pattern: /\bsnum\b/u,
+    reason: "expected text is likely sem"
+  },
+  {
+    name: "known Portuguese corruption cumprems",
+    pattern: /\bcumprems\b/u,
+    reason: "expected text is likely cumprem"
+  }
+];
+
+function listFiles(dirPath) {
+  return readdirSync(dirPath, { withFileTypes: true }).flatMap((entry) => {
+    const entryPath = path.join(dirPath, entry.name);
+    if (entry.isDirectory()) {
+      return listFiles(entryPath);
+    }
+    return [entryPath];
+  });
+}
+
+function collectStringValues(node, keyPath = "", results = []) {
+  if (typeof node === "string") {
+    results.push({ keyPath, value: node });
+    return results;
+  }
+
+  if (Array.isArray(node)) {
+    node.forEach((item, index) => {
+      collectStringValues(item, `${keyPath}[${index}]`, results);
+    });
+    return results;
+  }
+
+  if (node && typeof node === "object") {
+    for (const [key, value] of Object.entries(node)) {
+      collectStringValues(value, keyPath ? `${keyPath}.${key}` : key, results);
+    }
+  }
+
+  return results;
+}
+
+function validateTextEncodingAndMojibake() {
+  const jsonFiles = textQualityRoots
+    .flatMap((root) => listFiles(root))
+    .filter((filePath) => filePath.endsWith(".json"));
+
+  for (const filePath of jsonFiles) {
+    const data = readJson(filePath);
+    for (const { keyPath, value } of collectStringValues(data)) {
+      for (const { name, pattern, reason } of textQualityPatterns) {
+        if (pattern.test(value)) {
+          findings.push(
+            `Text quality issue in ${filePath}${keyPath ? ` at ${keyPath}` : ""}: ${name} (${reason}).`
+          );
+        }
+      }
+    }
+  }
+}
+
 function resolveStationCriteria(stationId, cycleId) {
   return stationCriteriaJson.byCycle?.[cycleId]?.[stationId]
     || stationCriteriaJson.default?.[stationId]
@@ -445,6 +543,8 @@ for (const locale of Object.keys(localizedCanvasDataJson)) {
   }
 }
 
+validateTextEncodingAndMojibake();
+
 if (findings.length > 0) {
   console.error("Method content validation failed:");
   for (const finding of findings) {

skills/localized-text-editing/SKILL.md

@@ -0,0 +1,48 @@
+---
+name: localized-text-editing
+description: Safely edit APIOps method and canvas labels containing localized or accented text without introducing mojibake, replacement characters, or Windows shell encoding damage. Use when changing src/data localized JSON, station labels, resource labels, canvas text, translated method content, or any non-ASCII method wording.
+---
+
+# Localized Text Editing
+
+Use this skill when editing APIOps method or canvas text that may contain non-ASCII characters, translated strings, smart punctuation, or locale-specific labels.
+
+## Rules
+
+- Treat `src/data/**/*.json` as UTF-8.
+- Avoid PowerShell heredocs and ad hoc shell text replacement for localized strings.
+- Prefer `apply_patch` for small exact edits when the target text is visible and uncorrupted.
+- For scripted edits, use Node with `readFileSync(file, "utf8")` and `writeFileSync(file, textOrJson)`.
+- When inserting non-ASCII text from a script, use Unicode escapes or preserve the exact existing value and replace only ASCII substrings.
+- Never rewrite whole localized files just to change a few translated strings unless the script is byte-safe and the diff is inspected.
+
+## Safe Workflow
+
+1. Inspect the current value with Node, not PowerShell text rendering, when encoding matters.
+2. Make the smallest edit that solves the content issue.
+3. Scan for known corruption patterns:
+
+```powershell
+node -e "const fs=require('fs'),path=require('path');function walk(d){return fs.readdirSync(d,{withFileTypes:true}).flatMap(e=>{const p=path.join(d,e.name);return e.isDirectory()?walk(p):[p]})}const pats=['Snum','snum','cumprems','n?o'];for(const f of walk('src/data').filter(f=>f.endsWith('.json'))){const s=fs.readFileSync(f,'utf8');for(const p of pats)if(s.includes(p))console.log(f+': '+p)}"
+```
+
+4. Run `npm.cmd test` on Windows.
+5. Review `git diff` for accidental `?`, mojibake, line-ending-only churn, and unrelated localized file rewrites.
+
+## Failure Patterns
+
+Treat these as likely corruption unless there is a clear, intentional reason:
+
+- `U+FFFD` replacement character
+- `U+00C3` or `U+00C2` mojibake leads
+- `U+00E2` followed by punctuation fragments from smart quotes or dashes
+- accent replacement like `n?o`, `prontid?o`, `integra??es`
+- known project typos: `Snum`, `snum`, `cumprems`
+
+## Recovery
+
+If an edit introduces many `?` characters or mojibake:
+
+1. Restore the affected file from the index or HEAD if those changes were yours.
+2. Reapply only the intended content changes using `apply_patch` or ASCII-only substring replacements.
+3. Rerun `npm.cmd test`.

skills/localized-text-editing/agents/openai.yaml

@@ -0,0 +1,5 @@
+interface:
+  display_name: Localized Text Editing
+  short_description: Edit localized APIOps text without encoding damage
+policy:
+  allow_implicit_invocation: true