Mirror the module tree in the docs folder by default

[anhnh2002]

Closes #125 (the hierarchical output part; OKF export is not included).

What changes

Pages now live in folders that follow the module tree instead of one flat folder. A module's page sits next to the folder that holds its sub-modules, and overview.md stays at the root:

docs/overview.md
docs/auth.md                 # links to auth/login.md, auth/session.md
docs/auth/login.md
docs/auth/session.md
docs/auth/session/store.md
docs/billing.md

A page never moves when sub-modules are added to it during a run. Module names stay unique across the wiki, so the module_tree.json key is still the page's filename stem and the tree format is unchanged.

The flat layout was originally chosen because small models kept getting relative links wrong. Current models handle nesting, so it is now the default, and --flat keeps the old layout for small models.

How

• codewiki/src/be/doc_layout.py (new) is the one place that maps modules to page paths, finds an existing page in either layout, and runs organize_docs after each run. That step moves pages an agent saved in the wrong folder and rewrites links between pages to correct relative paths. It is the safety net that makes nesting safe for weaker models.
• Prompts give each agent its own page path and every module's page path in the module tree outline. The sub-module tool reports the link to use from the parent page.
• Layout is recorded in metadata.json (generation_info.layout).
  • --update keeps the stored layout. Passing --flat against nested docs warns and is ignored.
  • Docs with no recorded layout predate this change, so they are treated as flat and stay flat.
• Updater:
  • Page lookup, removal, write sets, prompts and verdict parsing work with nested paths.
  • Folders left empty by a deleted page are removed.
• Viewers:
  • The GitHub Pages viewer embeds a module → path map (DOC_PATHS) and resolves links against the current page. A link that misses its page is matched by module name.
  • The FastAPI web app's navigation uses the same map. Its file route now rejects paths outside the docs folder.
• MCP:
  • Each entry in processing_order.json has a doc_path.
  • get_prompt accepts module_path.
  • close_session runs the same cleanup and records the layout.
  • The wiki-generator skill is updated.
• & in module names is now replaced with _. In a real run the agent shell-escaped it and wrote pages into a stray A_\&_B/ folder. This only affects newly clustered trees.

Testing

• tests/test_doc_layout.py (20 tests) covers:
  • path mapping and layout detection
  • moving misplaced pages and repairing links (relative paths, anchors, external links, code fences)
  • prompts in both layouts
  • nested generation and resume
  • updater pages
  • the editor creating nested folders while still rejecting paths that escape the docs folder
  • the viewer path map and the recorded layout
• The full suite passes, except test_clustering_skip::test_over_threshold_calls_llm_and_bad_response_falls_back, which also fails on main.
• tests/smoke_test_mcp.py passes.
• End-to-end run with claude-code / Haiku on a copy of updater/ + agent_tools/:
  • All 27 modules got a page at the expected path, up to four folders deep.
  • 249 of 257 links between pages resolve. The other 8 are example links in prose (the documented code is about link handling) and one link to a module name the model made up.

Known gaps

• Weak models sometimes write extra pages that aren't modules (for example README.md) inside module folders. They are not in the module tree, so the navigation doesn't list them, as in the flat layout.
• There is no automatic migration of existing flat docs to the nested layout: --update keeps them flat. Regenerating without --update reuses the existing pages and moves them into folders.