feat(config): migrate config format from JSON to TOML with automatic upgrade path#1317
Open
CaptainMirage wants to merge 14 commits into
Open
feat(config): migrate config format from JSON to TOML with automatic upgrade path#1317CaptainMirage wants to merge 14 commits into
CaptainMirage wants to merge 14 commits into
Conversation
Adds the `toml = "0.8"` dependency to Cargo.toml as the first step of
the JSON -> TOML config migration. No behaviour changes in this commit --
the existing flat Config struct and all callers are completely untouched.
src/config.rs:
- Add Serialize to ScriptId derive (required for the migration write
path in the next commit)
- Append TOML intermediate structs at the bottom of the file:
- TomlRelay: maps the [relay] section, covers all relay/batching/
blacklist/timeout fields
- TomlNetwork: maps the [network] section with Default impl, includes
block_stun field (added in v1.9.28 therealaleph#1115, absent from original brief)
- TomlScan: maps the [scan] section with Default impl
- TomlLogging: maps the [logging] section with Default impl
- TomlConfig: root document struct, composes all sections plus
existing ExitNodeConfig and FrontingGroup
These structs are only used inside Config::load_toml and the JSON->TOML
migration writer added in subsequent commits. Both paths produce a flat
Config via From<TomlConfig> so nothing outside config.rs needs to change
…versions Add From<TomlConfig> for Config to flatten the grouped TOML representation into the existing flat Config struct that the rest of the codebase uses. Add From<&Config> for TomlConfig for the migration write path -- takes a reference so the flat Config can still be returned after the TOML is written. Add Config::load_toml(path) to read a .toml file, deserialize into TomlConfig, convert to flat Config, and run validate(). Add ParseToml variant to ConfigError wrapping toml::de::Error. Call sites unchanged, Config::load still reads JSON only. TOML loading will be wired in the next commit when i confirm its working properly
Replace Config::load with a dispatch function that routes to Config::load_toml or a new private load_json_and_migrate helper based on the file extension. On a .json path, parse JSON into a flat Config, convert to TomlConfig via From<&Config>, serialize with toml::to_string_pretty, and write the result alongside the JSON file with a .toml extension. Write failure is non-fatal -- the in-memory Config is returned so the proxy keeps running. Emits a tracing::warn so users know migration happened and can delete the old file. On a .toml path, delegates to Config::load_toml from the previous commit. On an unknown or missing extension, tries TOML then JSON; surfaces the TOML error on double failure. No call sites changed yet, Config::load signature is unchanged so callers pick up the new behaviour automatically.
… as TOML ui.rs save_config: replace serde_json::to_string_pretty(ConfigWire) with toml::to_string_pretty(TomlConfig) so the UI writes config.toml on Save. ConfigWire is unchanged, it is still used for the loading display path. android_jni.rs: startProxy accepted only JSON strings from the Kotlin app. Extend to try JSON first then fall back to TOML so the native layer handles both formats. Add TomlConfig to imports. Update the doc comment and error message to be format-agnostic. Three UI strings updated: "config.json" -> "config.toml" in the listen_host hint, the CA-remove log line, and the CA-remove hover text.
config_path() now returns data_dir/config.toml. Add json_config_path() returning data_dir/config.json for use by resolve_config_path when detecting legacy configs that need auto-migration. resolve_config_path updated to check toml before json in both the user-data dir and cwd. JSON hits still return the json path so Config::load's migration logic fires naturally on the next startup. Falls back to data_dir/config.toml for new users so error messages and Save-config operations point to the right place. ui.rs loading block simplified to a single resolve_config_path(None) call, removing the duplicate two-path check. Behaviour is identical but the path discovery logic now lives in one place. main.rs help text corrected: old string claimed default was ./config.json (cwd) but the actual default has always been data_dir via resolve_config_path. will soon Update to show the real runtime path dynamically
… refs Delete all five config.*.example.json files and replace them with config.*.example.toml equivalents using the new grouped TOML schema. Update the validation error message in config.rs that named config.json directly (socks5_port/listen_port conflict) to reference config.toml. Update the startup error message in main.rs that directed users to copy config.example.json to reference config.example.toml.
Add toml_tests module in config.rs covering: - Minimal [relay]-only TOML parses and validates correctly - Omitting [network] applies all defaults (google_ip, listen_port, etc.) - [exit_node] table round-trips fields correctly - [[fronting_groups]] array-of-tables parses multiple groups - [network.hosts] subtable populates the hosts HashMap - script_id as a TOML array round-trips through script_ids_resolved() - Config::load dispatches to TOML path when given a .toml extension - JSON migration: Config::load on a .json file writes config.toml alongside it, and the written file round-trips back to an equivalent Config with matching mode, auth_key, script_id, and listen_port
Replaces config.json with config.toml in all user-facing documentation, example code blocks, launcher scripts, and asset READMEs following the JSON → TOML config migration. Intentionally left unchanged: - Android ConfigStore.kt (own internal JSON storage, unrelated) - Native.kt JNI bridge (passes JSON string to Rust core by design) - src/data_dir.rs and src/config.rs migration logic (must reference the legacy filename) - Changelog entries for versions prior to the migration - scan_config.json references (separate file, not the main config) will eventually update android version too, maybe
…ig.toml (i forgot to save these)
Config::load fires before init_logging in the normal serve path, so the
tracing::warn! in load_json_and_migrate was silently dropped - the
subscriber didn't exist yet. Fix by calling init_logging("warn") before
Config::load so the migration notice is visible. The second init_logging
call with config.log_level is a harmless no-op (try_init semantics);
RUST_LOG is still respected because EnvFilter checks the env var first,
and "warn" is already the compiled default for log_level anyway
this adds no overhead, the subscriber is registered once; the second call is a no-op.
…lised
In my infinite wisdom, my previous commit attempted to fix the silent
migration warning by calling init_logging("warn") before Config::load.
The real problem was that load_json_and_migrate fires tracing::warn! from inside
Config::load, but the subscriber can only be initialised after Config::load
returns, because the user's log_level lives inside the config that hasn't
been read yet. Classic chicken-and-egg. Locking to "warn" early meant the
second init_logging(&config.log_level) call was a silent no-op (try_init
semantics), so users lost their configured log level entirely.
Fix: instead of logging directly inside load_json_and_migrate, collect
the warning as an Option<String> and return it alongside the Config up
the call chain. main.rs logs it with tracing::warn! after
init_logging(&config.log_level) — one subscriber, correct level,
migration warning visible with proper timestamp and severity.
Config::load return type changes from Result<Self> to
Result<(Self, Option<String>)>; load_toml paths return None, the JSON
migration path returns Some(message). Test call sites updated with .0
to destructure the tuple.
Config::load now returns Result<(Self, Option<String>)> — the Ok arm in load_form's match was still destructuring as Ok(c). Updated to Ok((c, _)) to discard the migration warning (the UI has no logging path for it at load time).
…gration Merge remote-tracking branch 'upstream/main' into feat/toml-config-migration
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Migrates the user-facing configuration format from JSON to TOML across the
entire codebase. Existing
config.jsonfiles are automatically translated toconfig.tomlon the next startup — no manual action required. The JSON parsepath is kept for the migration step and for the Android JNI layer (which
receives config as a string from the Kotlin app).
Changes
Core config loading (
src/config.rs)toml = "0.8"to[dependencies]Serializederive toScriptId(needed for migration write path)TomlRelay,TomlNetwork,TomlScan,TomlLogging,TomlConfig— each section maps to a grouped TOML table.Defaultimpls on each section struct call the existing named defaultfunctions so omitting a section in the file gives the same defaults as before
From<TomlConfig> for Configflattens the grouped representation back intothe existing flat
Configstruct. All call sites are unaffectedFrom<&Config> for TomlConfigbuilds the grouped form for migration writesand the UI save path
Config::load_toml(path)— reads and parses a.tomlfile, converts toflat
Config, runsvalidate()Config::loadupdated with a format-dispatch decision tree:.tomlextension routes to
load_toml,.jsonroutes toload_json_and_migrate,unknown extension tries TOML first then JSON
Config::load_json_and_migrate— parses JSON, writes a.tomlequivalentalongside the JSON file, returns the message to emit as a warn after the
subscriber is initialised (see logging note below)
ParseToml(#[from] toml::de::Error)added toConfigErrorvalidate()port conflict message updated to referenceconfig.tomlLogging chicken-and-egg fix (
src/main.rs,src/config.rs)The migration warning fires inside
Config::load, but the tracing subscriberis initialised after
Config::loadreturns becauselog_levellives in theconfig. Calling
init_logging("warn")before load and a secondinit_logging(&config.log_level)after was silently locking all users to"warn"—tracing'stry_initonly lets one subscriber register.Fix:
load_json_and_migratecollects the warning message withformat!()andreturns it as
Option<String>alongside theConfig.Config::loadreturntype changed to
Result<(Self, Option<String>), ConfigError>.main.rsdestructures the result, calls
init_logging(&config.log_level), then emitsthe warning if present. One subscriber, correct user-configured log level,
warning visible.
src/data_dir.rsconfig_path()now returnsdata_dir/config.tomljson_config_path()returningdata_dir/config.jsonfor themigration detection path
resolve_config_path()updated: checksdata_dir/config.tomlfirst, thendata_dir/config.json, then./config.toml, then./config.json. Fallsback to
data_dir/config.tomlfor new users so error messages and Saveoperations point to the right place
src/bin/ui.rssave_configchanged fromserde_json::to_string_pretty(&ConfigWire::from(cfg))to
toml::to_string_pretty(&TomlConfig::from(cfg)). Saves are now TOMLresolve_config_path(None)call,removing duplicate two-path check logic that
resolve_config_pathnow ownsConfig::loadcall site updated to destructure the new(Config, Option<String>)return typeconfig.jsontoconfig.toml(listen_hosthint, CA-remove log line, CA-remove hover text)
src/android_jni.rsTomlConfigadded to importsstartProxynow tries JSON parsing first (backward compat with existingKotlin app), falls back to TOML if JSON fails
Example configs
All five JSON example files deleted and replaced with
.tomlequivalentsusing the new grouped schema:
config.example.tomlconfig.direct.example.tomlconfig.full.example.tomlconfig.exit-node.example.tomlconfig.fronting-groups.example.tomlDocumentation
All user-facing references to
config.jsonupdated toconfig.tomlacrossSF_README.md(English and Farsi),docs/guide.md,docs/guide.fa.md,docs/fronting-groups.md,assets/exit_node/README.md(English and Farsi),assets/cloudflare/README.md(English and Farsi),assets/launchers/run.bat,assets/openwrt/mhrv-rs.init, and the Apps Script files. JSON config snippetsin docs replaced with TOML equivalents using correct section syntax.
Migration path for existing users
No action required. On the first run after update:
resolve_config_pathfindsconfig.jsonin the data dir or cwdConfig::loadsees the.jsonextension and callsload_json_and_migrateconfig.tomlis written alongside it, a warning islogged: "Found legacy config.json. Translated to config.toml automatically.
config.json has been left in place but will no longer be read. You can
delete it."
config.tomlis found first and loaded directlyWrite failure during migration is non-fatal — the proxy continues running
from the in-memory config.
Tests
config::toml_tests:[relay]-only config parses and validates[network]applies all defaults correctly[exit_node]table round-trips all fields[[fronting_groups]]array-of-tables parses multiple entries[network.hosts]subtable populates the hosts HashMapscript_idas a TOML array round-trips throughscript_ids_resolved()Config::loaddispatches correctly on.tomlextensionconfig.tomlalongside and the result round-tripsrt_tests::round_trip_all_current_fieldsandrt_tests::round_trip_minimal_fields_only— both callConfig::loadon.jsontemp files which now trigger migration; the written.tomlis nowcleaned up alongside the
.jsonWhy TOML instead of JSON
Here's the Why TOML section rewritten as a short paragraph instead of bullet points:
Why TOML instead of JSON
JSON was never designed for config files — it has no comment syntax, requires quoting every key, and a flat structure that stops making sense once you have 30+ fields across multiple logical groups. TOML was built specifically for this. Named sections (
[relay],[network],[exit_node]) make it obvious where each setting belongs,#comments survive round-trips so example files can actually explain themselves, and there's no trailing-comma rule to trip over when editing by hand. Parse speed is a non-argument — it's read once at startup, the difference is microseconds. It's also the de facto standard in the Rust ecosystem;Cargo.tomlbeing the most obvious example, so anyone already working in this codebase already knows the format.