Copy preferences into the test environment by davidanthoff · Pull Request #121 · JuliaTesting/TestEnv.jl

davidanthoff · 2026-08-19T23:33:21Z

Finishes the work started in #97 (whose commit is cherry-picked here, so @NHDaly keeps authorship of it) and closes #59.

Pkg.test copies the merged preferences into the sandbox it builds; TestEnv did not, so packages whose tests are configured through Preferences.jl pass under ]test but not under TestEnv.activate(). This ports Pkg's logic across.

What changed on top of #97

Extended to every version directory that can support it. Support Preferences in TestEnv: Copy the preferences into the TestEnv. #97 patched src/julia-1.9 only, which covers Julia 1.9–1.10. The lookup now lives in common.jl as sandbox_preferences and is applied in julia-1.8, julia-1.9, julia-1.11, julia-1.12 and julia-1.13.
The do-block form is fixed too. TestEnv.activate(f, pkg) delegates to Pkg.Operations.sandbox, which has taken a preferences= keyword since Julia 1.8 but was never passed one — so that form dropped preferences even with the set form fixed.
Empty-preferences guard. Base.get_preferences() returns an empty Dict, not nothing, so the !== nothing check always fired and always wrote an empty preferences file. It now guards on !isempty.
Dropped the source_path local that shadowed Pkg.Operations.source_path, qualified copy!(Base.LOAD_PATH, ...) in the restore path, and answered the # TODO: should we separately import TOML? in a comment — no, TestEnv supports Julia 1.0, which predates the TOML stdlib, so Pkg.TOML it is.

Why not Julia 1.7 and earlier

Base.get_preferences only grew its no-argument, whole-set form in Julia 1.8; before that it (and collect_preferences) is scoped to a single package UUID, so there is no way to collect the full merged set to copy across without reimplementing Base's merge. Pkg.Operations.sandbox likewise only gained its preferences= keyword in 1.8. Older versions keep today's behaviour, and the README now says so.

Tests

Three offline fixture packages under test/preferences/ — no registry dependencies, so they instantiate without network:

PrefsTopLevel — preferences at the package root, no test/Project.toml
PrefsTestDir — a test/ environment whose preferences shadow the package root's, with one key only set at the root to check the merge order
PrefsNone — no preferences at all, to cover the isempty guard

Both activate forms are covered. Verified that all of these fail on main and pass here.

Drive-by fix

test/activate_set.jl's precompile testset guarded its finally with isdefined(@__MODULE__, :orig_load_path) — those are locals inside the testset, not module globals, so the guard was never true and LOAD_PATH/DEPOT_PATH leaked into every later testset. Restoring them unconditionally is needed for the new tests to mean anything.

Verification

Full suite run locally against Julia 1.6.7, 1.8.5, 1.9.4, 1.11.9, 1.12.7 and 1.13.0-rc3 — all pass.

🤖 Generated with Claude Code

This code is based on what Pkg does, as identified here: #59 (comment) - Check if there are LocalPreferences.toml in test/ - If not, check in top-level package path - Copy whatever LocalPreferences.toml you find into the tmp env.

Move the preference lookup into `common.jl` as `sandbox_preferences`, shared by both `activate` forms, and apply it to the julia-1.8, 1.9, 1.11, 1.12 and 1.13 directories rather than just julia-1.9. Julia 1.7 and earlier are left alone: `Base.get_preferences` there only exposes preferences per package UUID, so there is no way to collect the whole merged set to copy across. Along the way: - `Base.get_preferences()` returns an empty `Dict` rather than `nothing` when there are no preferences, so the old `!== nothing` guard always fired and always wrote an empty preferences file. Guard on `!isempty` instead. - Drop the `source_path` local, which shadowed `Pkg.Operations.source_path`. - Qualify `copy!(Base.LOAD_PATH, ...)` in the restore path. - Answer the `# TODO: should we separately import TOML?` in a comment: no -- TestEnv supports Julia 1.0, which predates the `TOML` stdlib. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`TestEnv.activate(f, pkg)` delegates to `Pkg.Operations.sandbox`, which has taken a `preferences=` keyword since Julia 1.8 but was never passed one, so that form dropped preferences even with the set form fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The `isdefined(@__MODULE__, :orig_load_path)` guards were never true -- those are locals inside the testset, not module globals -- so the `finally` block silently did nothing and both globals leaked into every later testset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three offline fixture packages under test/preferences/ (no registry deps, so they instantiate without network): preferences at the package root only, in `test/` shadowing the package root, and none at all. Both `activate` forms are covered, plus the negative case that no preferences means no preferences file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

codecov · 2026-08-19T23:35:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.36%. Comparing base (1766d3f) to head (b4281de).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #121      +/-   ##
==========================================
+ Coverage   81.69%   83.36%   +1.67%     
==========================================
  Files          34       34              
  Lines         945     1040      +95     
==========================================
+ Hits          772      867      +95     
  Misses        173      173

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

oxinabox · 2026-09-07T08:18:25Z

    tmp = mktempdir()
    tmp_project = projectfile_path(tmp)
    tmp_manifest = manifestfile_path(tmp)
+    tmp_preferences = joinpath(tmp, first(Base.preferences_names))


is this correct?

Yes. Base.preferences_names is ("JuliaLocalPreferences.toml", "LocalPreferences.toml"), and Base.collect_preferences loops over that tuple and breaks at the first file that exists — so the first name is the one that actually wins, and writing it means a stray LocalPreferences.toml can't shadow what we put there. It's also verbatim what Pkg.Operations.sandbox does. I've added a comment saying so, since it clearly isn't obvious from the code.

oxinabox · 2026-09-07T08:19:08Z

+    # Copy the preferences over too, as `Pkg.test` does. We reach for `Pkg.TOML` rather
+    # than the `TOML` stdlib because TestEnv supports Julia 1.0, which predates it.


but this file is only used in Julia 1.11+

You're right, that reason is nonsense in a file that only ever loads on 1.11+. Pkg.TOML is still the right call, but for a reason the comment failed to state: TestEnv has one Project.toml shared by all eleven version directories with julia = "1" compat, and TOML only became a stdlib in 1.6 — so listing it in [deps] would make the package unloadable on 1.0–1.5. Pkg is already a dependency and does import TOML internally on every version from 1.8 up, so Pkg.TOML gets us the stdlib module for free. Comment now says that, in all five copies.

oxinabox · 2026-09-07T08:19:33Z

+    # Copy the preferences over too, as `Pkg.test` does. We reach for `Pkg.TOML` rather
+    # than the `TOML` stdlib because TestEnv supports Julia 1.0, which predates it.


but this file is only used in 1.12

Same as above — fixed here too. The justification now points at the shared Project.toml rather than at this file's own version floor.

oxinabox · 2026-09-07T08:21:52Z

+# `sandbox_preferences` helper in `src/julia-1.8/common.jl` and its siblings.
+if VERSION >= v"1.8-"
+    @testset "preferences.jl" begin
+        fixture(name) = joinpath(@__DIR__, "preferences", name)


weird indirection, why is this here?

Gone — inlined to joinpath(@__DIR__, "preferences", "PrefsTopLevel"), which is what test/activate_set.jl already does for the sources/ fixtures.

oxinabox · 2026-09-07T08:22:07Z

+        # `Pkg` -- and hence TestEnv -- writes to the first of `Base.preferences_names`
+        preferences_file(project=Base.active_project()) =
+            joinpath(dirname(project), first(Base.preferences_names))
+
+        # The preferences the sandbox itself carries, read straight off disk so the
+        # assertion does not depend on whatever `LOAD_PATH` happens to look like.
+        sandbox_preferences(name) = Pkg.TOML.parsefile(preferences_file())[name]


weird indirections, why are these here?

Also gone, and they were worse than merely redundant: preferences_file's project parameter was never once passed, and the local sandbox_preferences shadowed the name of the real function in common.jl while meaning something different. Each testset now spells out prefs_file = joinpath(dirname(Base.active_project()), first(Base.preferences_names)) and reads it with Pkg.TOML.parsefile. The three UUID constants were used exactly once each, so they went the same way.

NHDaly and others added 6 commits August 19, 2026 15:59

Document preferences support and bump to 1.104.0

31c887f

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

davidanthoff mentioned this pull request Aug 19, 2026

Support Preferences in TestEnv: Copy the preferences into the TestEnv. #97

Draft

3 tasks

Merge branch 'main' into preferences

5386d88

oxinabox reviewed Sep 7, 2026

View reviewed changes

oxinabox requested changes Sep 7, 2026

View reviewed changes

Address feedback

b4281de

Navigation Menu

Uh oh!

Copy preferences into the test environment - #121

Copy preferences into the test environment#121
davidanthoff wants to merge 8 commits into
mainfrom
preferences

davidanthoff commented Aug 19, 2026

Uh oh!

codecov Bot commented Aug 19, 2026 •

edited

Loading

Uh oh!

oxinabox Sep 7, 2026

Uh oh!

davidanthoff Sep 8, 2026

Uh oh!

oxinabox Sep 7, 2026

Uh oh!

davidanthoff Sep 8, 2026

Uh oh!

oxinabox Sep 7, 2026

Uh oh!

davidanthoff Sep 8, 2026

Uh oh!

oxinabox Sep 7, 2026

Uh oh!

davidanthoff Sep 8, 2026

Uh oh!

oxinabox Sep 7, 2026

Uh oh!

davidanthoff Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# Copy the preferences over too, as `Pkg.test` does. We reach for `Pkg.TOML` rather
		# than the `TOML` stdlib because TestEnv supports Julia 1.0, which predates it.

Uh oh!

Conversation

davidanthoff commented Aug 19, 2026

What changed on top of #97

Why not Julia 1.7 and earlier

Tests

Drive-by fix

Verification

Uh oh!

codecov Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Aug 19, 2026 •

edited

Loading