Skip to content

feat(config): migrate config format from JSON to TOML with automatic upgrade path#1317

Open
CaptainMirage wants to merge 14 commits into
therealaleph:mainfrom
CaptainMirage:feat/toml-config-migration
Open

feat(config): migrate config format from JSON to TOML with automatic upgrade path#1317
CaptainMirage wants to merge 14 commits into
therealaleph:mainfrom
CaptainMirage:feat/toml-config-migration

Conversation

@CaptainMirage
Copy link
Copy Markdown
Contributor

What this does

Migrates the user-facing configuration format from JSON to TOML across the
entire codebase. Existing config.json files are automatically translated to
config.toml on the next startup — no manual action required. The JSON parse
path is kept for the migration step and for the Android JNI layer (which
receives config as a string from the Kotlin app).

Changes

Core config loading (src/config.rs)

  • Added toml = "0.8" to [dependencies]
  • Added Serialize derive to ScriptId (needed for migration write path)
  • Defined intermediate TOML structs: TomlRelay, TomlNetwork, TomlScan,
    TomlLogging, TomlConfig — each section maps to a grouped TOML table.
    Default impls on each section struct call the existing named default
    functions so omitting a section in the file gives the same defaults as before
  • From<TomlConfig> for Config flattens the grouped representation back into
    the existing flat Config struct. All call sites are unaffected
  • From<&Config> for TomlConfig builds the grouped form for migration writes
    and the UI save path
  • Config::load_toml(path) — reads and parses a .toml file, converts to
    flat Config, runs validate()
  • Config::load updated with a format-dispatch decision tree: .toml
    extension routes to load_toml, .json routes to load_json_and_migrate,
    unknown extension tries TOML first then JSON
  • Config::load_json_and_migrate — parses JSON, writes a .toml equivalent
    alongside the JSON file, returns the message to emit as a warn after the
    subscriber is initialised (see logging note below)
  • ParseToml(#[from] toml::de::Error) added to ConfigError
  • validate() port conflict message updated to reference config.toml

Logging chicken-and-egg fix (src/main.rs, src/config.rs)

The migration warning fires inside Config::load, but the tracing subscriber
is initialised after Config::load returns because log_level lives in the
config. Calling init_logging("warn") before load and a second
init_logging(&config.log_level) after was silently locking all users to
"warn"tracing's try_init only lets one subscriber register.

Fix: load_json_and_migrate collects the warning message with format!() and
returns it as Option<String> alongside the Config. Config::load return
type changed to Result<(Self, Option<String>), ConfigError>. main.rs
destructures the result, calls init_logging(&config.log_level), then emits
the warning if present. One subscriber, correct user-configured log level,
warning visible.

src/data_dir.rs

  • config_path() now returns data_dir/config.toml
  • Added json_config_path() returning data_dir/config.json for the
    migration detection path
  • resolve_config_path() updated: checks data_dir/config.toml first, then
    data_dir/config.json, then ./config.toml, then ./config.json. Falls
    back to data_dir/config.toml for new users so error messages and Save
    operations point to the right place

src/bin/ui.rs

  • save_config changed from serde_json::to_string_pretty(&ConfigWire::from(cfg))
    to toml::to_string_pretty(&TomlConfig::from(cfg)). Saves are now TOML
  • Loading block simplified to a single resolve_config_path(None) call,
    removing duplicate two-path check logic that resolve_config_path now owns
  • Config::load call site updated to destructure the new
    (Config, Option<String>) return type
  • Three UI strings updated from config.json to config.toml (listen_host
    hint, CA-remove log line, CA-remove hover text)

src/android_jni.rs

  • TomlConfig added to imports
  • startProxy now tries JSON parsing first (backward compat with existing
    Kotlin app), falls back to TOML if JSON fails
  • Doc comment updated to be format-agnostic

Example configs

All five JSON example files deleted and replaced with .toml equivalents
using the new grouped schema:

  • config.example.toml
  • config.direct.example.toml
  • config.full.example.toml
  • config.exit-node.example.toml
  • config.fronting-groups.example.toml

Documentation

All user-facing references to config.json updated to config.toml across
SF_README.md (English and Farsi), docs/guide.md, docs/guide.fa.md,
docs/fronting-groups.md, assets/exit_node/README.md (English and Farsi),
assets/cloudflare/README.md (English and Farsi), assets/launchers/run.bat,
assets/openwrt/mhrv-rs.init, and the Apps Script files. JSON config snippets
in docs replaced with TOML equivalents using correct section syntax.

Migration path for existing users

No action required. On the first run after update:

  1. resolve_config_path finds config.json in the data dir or cwd
  2. Config::load sees the .json extension and calls load_json_and_migrate
  3. JSON is parsed, a config.toml is written alongside it, a warning is
    logged: "Found legacy config.json. Translated to config.toml automatically.
    config.json has been left in place but will no longer be read. You can
    delete it."
  4. On all subsequent runs, config.toml is found first and loaded directly

Write failure during migration is non-fatal — the proxy continues running
from the in-memory config.

Tests

  • 8 new tests in config::toml_tests:
    • Minimal [relay]-only config parses and validates
    • Omitting [network] applies all defaults correctly
    • [exit_node] table round-trips all fields
    • [[fronting_groups]] array-of-tables parses multiple entries
    • [network.hosts] subtable populates the hosts HashMap
    • script_id as a TOML array round-trips through script_ids_resolved()
    • Config::load dispatches correctly on .toml extension
    • JSON migration writes config.toml alongside and the result round-trips
  • Fixed orphaned temp files in rt_tests::round_trip_all_current_fields and
    rt_tests::round_trip_minimal_fields_only — both call Config::load on
    .json temp files which now trigger migration; the written .toml is now
    cleaned up alongside the .json
  • Full test suite: 248 tests, 0 failures

Why TOML instead of JSON

Here's the Why TOML section rewritten as a short paragraph instead of bullet points:


Why TOML instead of JSON

JSON was never designed for config files — it has no comment syntax, requires quoting every key, and a flat structure that stops making sense once you have 30+ fields across multiple logical groups. TOML was built specifically for this. Named sections ([relay], [network], [exit_node]) make it obvious where each setting belongs, # comments survive round-trips so example files can actually explain themselves, and there's no trailing-comma rule to trip over when editing by hand. Parse speed is a non-argument — it's read once at startup, the difference is microseconds. It's also the de facto standard in the Rust ecosystem; Cargo.toml being the most obvious example, so anyone already working in this codebase already knows the format.

Adds the `toml = "0.8"` dependency to Cargo.toml as the first step of
the JSON -> TOML config migration. No behaviour changes in this commit --
the existing flat Config struct and all callers are completely untouched.

src/config.rs:
- Add Serialize to ScriptId derive (required for the migration write
  path in the next commit)
- Append TOML intermediate structs at the bottom of the file:
  - TomlRelay: maps the [relay] section, covers all relay/batching/
    blacklist/timeout fields
  - TomlNetwork: maps the [network] section with Default impl, includes
    block_stun field (added in v1.9.28 therealaleph#1115, absent from original brief)
  - TomlScan: maps the [scan] section with Default impl
  - TomlLogging: maps the [logging] section with Default impl
  - TomlConfig: root document struct, composes all sections plus
    existing ExitNodeConfig and FrontingGroup

These structs are only used inside Config::load_toml and the JSON->TOML
migration writer added in subsequent commits. Both paths produce a flat
Config via From<TomlConfig> so nothing outside config.rs needs to change
…versions

Add From<TomlConfig> for Config to flatten the grouped TOML representation
into the existing flat Config struct that the rest of the codebase uses.
Add From<&Config> for TomlConfig for the migration write path -- takes a
reference so the flat Config can still be returned after the TOML is written.
Add Config::load_toml(path) to read a .toml file, deserialize into TomlConfig,
convert to flat Config, and run validate(). Add ParseToml variant to
ConfigError wrapping toml::de::Error.

Call sites unchanged, Config::load still reads JSON only. TOML loading
will be wired in the next commit when i confirm its working properly
Replace Config::load with a dispatch function that routes to Config::load_toml
or a new private load_json_and_migrate helper based on the file extension.

On a .json path, parse JSON into a flat Config, convert to TomlConfig via
From<&Config>, serialize with toml::to_string_pretty, and write the result
alongside the JSON file with a .toml extension. Write failure is non-fatal
-- the in-memory Config is returned so the proxy keeps running. Emits a
tracing::warn so users know migration happened and can delete the old file.

On a .toml path, delegates to Config::load_toml from the previous commit.
On an unknown or missing extension, tries TOML then JSON; surfaces the TOML
error on double failure.

No call sites changed yet, Config::load signature is unchanged so callers
pick up the new behaviour automatically.
… as TOML

ui.rs save_config: replace serde_json::to_string_pretty(ConfigWire) with
toml::to_string_pretty(TomlConfig) so the UI writes config.toml on Save.
ConfigWire is unchanged, it is still used for the loading display path.

android_jni.rs: startProxy accepted only JSON strings from the Kotlin app.
Extend to try JSON first then fall back to TOML so the native layer handles
both formats. Add TomlConfig to imports. Update the doc comment and error
message to be format-agnostic.

Three UI strings updated: "config.json" -> "config.toml" in the listen_host
hint, the CA-remove log line, and the CA-remove hover text.
config_path() now returns data_dir/config.toml. Add json_config_path()
returning data_dir/config.json for use by resolve_config_path when
detecting legacy configs that need auto-migration.

resolve_config_path updated to check toml before json in both the
user-data dir and cwd. JSON hits still return the json path so
Config::load's migration logic fires naturally on the next startup.
Falls back to data_dir/config.toml for new users so error messages and
Save-config operations point to the right place.

ui.rs loading block simplified to a single resolve_config_path(None)
call, removing the duplicate two-path check. Behaviour is identical
but the path discovery logic now lives in one place.

main.rs help text corrected: old string claimed default was ./config.json
(cwd) but the actual default has always been data_dir via
resolve_config_path. will soon Update to show the real runtime path dynamically
… refs

Delete all five config.*.example.json files and replace them with
config.*.example.toml equivalents using the new grouped TOML schema.

Update the validation error message in config.rs that named config.json
directly (socks5_port/listen_port conflict) to reference config.toml.

Update the startup error message in main.rs that directed users to copy
config.example.json to reference config.example.toml.
Add toml_tests module in config.rs covering:
- Minimal [relay]-only TOML parses and validates correctly
- Omitting [network] applies all defaults (google_ip, listen_port, etc.)
- [exit_node] table round-trips fields correctly
- [[fronting_groups]] array-of-tables parses multiple groups
- [network.hosts] subtable populates the hosts HashMap
- script_id as a TOML array round-trips through script_ids_resolved()
- Config::load dispatches to TOML path when given a .toml extension
- JSON migration: Config::load on a .json file writes config.toml
  alongside it, and the written file round-trips back to an equivalent
  Config with matching mode, auth_key, script_id, and listen_port
Replaces config.json with config.toml in all user-facing documentation,
example code blocks, launcher scripts, and asset READMEs following the
JSON → TOML config migration.

Intentionally left unchanged:
- Android ConfigStore.kt (own internal JSON storage, unrelated)
- Native.kt JNI bridge (passes JSON string to Rust core by design)
- src/data_dir.rs and src/config.rs migration logic (must reference
  the legacy filename)
- Changelog entries for versions prior to the migration
- scan_config.json references (separate file, not the main config)

will eventually update android version too, maybe
Config::load fires before init_logging in the normal serve path, so the
tracing::warn! in load_json_and_migrate was silently dropped - the
subscriber didn't exist yet. Fix by calling init_logging("warn") before
Config::load so the migration notice is visible. The second init_logging
call with config.log_level is a harmless no-op (try_init semantics);
RUST_LOG is still respected because EnvFilter checks the env var first,
and "warn" is already the compiled default for log_level anyway

this adds no overhead, the subscriber is registered once; the second call is a no-op.
…lised

In my infinite wisdom, my previous commit attempted to fix the silent
migration warning by calling init_logging("warn") before Config::load.
The real problem was that load_json_and_migrate fires tracing::warn! from inside
Config::load, but the subscriber can only be initialised after Config::load
returns, because the user's log_level lives inside the config that hasn't
been read yet. Classic chicken-and-egg. Locking to "warn" early meant the
second init_logging(&config.log_level) call was a silent no-op (try_init
semantics), so users lost their configured log level entirely.

Fix: instead of logging directly inside load_json_and_migrate, collect
the warning as an Option<String> and return it alongside the Config up
the call chain. main.rs logs it with tracing::warn! after
init_logging(&config.log_level) — one subscriber, correct level,
migration warning visible with proper timestamp and severity.

Config::load return type changes from Result<Self> to
Result<(Self, Option<String>)>; load_toml paths return None, the JSON
migration path returns Some(message). Test call sites updated with .0
to destructure the tuple.
Config::load now returns Result<(Self, Option<String>)> — the Ok arm
in load_form's match was still destructuring as Ok(c). Updated to
Ok((c, _)) to discard the migration warning (the UI has no logging
path for it at load time).
…gration

Merge remote-tracking branch 'upstream/main' into feat/toml-config-migration
@github-actions github-actions Bot added the type: feature feat: PR — auto-applied by release-drafter label May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feature feat: PR — auto-applied by release-drafter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant