SG-43461 SG-43644 Migrate host base and other support - #1108
Merged
Conversation
Adds requirements/any/ for Python-version-independent vendor zips and teaches python/tank_vendor/__init__.py to auto-discover and load them alongside the existing pkgs.zip. Drops in flow_data_sdk-beta.zip as the first such vendor. The loader refactor extracts the existing pkgs.zip init into a reusable _load_packages_from_zip helper. Shared zips load after pkgs.zip so per-version pins win on name collision; collisions warn and skip rather than overwrite. Per-package import failures continue to warn-and-continue (the SDK uses 3.10+ syntax, so it'll simply be absent on 3.7/3.9 instead of breaking import tank_vendor). Includes a small _patch_flow_data_sdk_version workaround for an upstream bug: the SDK's _version.py queries importlib.metadata.version( "adsk-flow-data") but the published wheel's distribution name is "flow-data-sdk", so SDK_VERSION otherwise falls back to "local_dev" even with .dist-info present. The patch is a self-disabling no-op once upstream is fixed. Tests cover the new package via PACKAGES_TO_TEST (3.10+ gated) plus a TestFlowDataSDK class with a dist-info canary that catches future regressions in the zip's metadata packaging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Flow Data SDK previously hardcoded the wrong distribution name in
flow_data_sdk/base/_version.py ("adsk-flow-data" instead of the actual
published name "flow-data-sdk"), so importlib.metadata.version() always
raised PackageNotFoundError and SDK_VERSION fell back to "local_dev"
even with .dist-info present in our shared zip.
The new SDK zip ships the upstream one-line fix — _version.py now
queries the correct name and SDK_VERSION resolves to the real version
on its own. The local workaround patch can go.
Removes:
- _patch_flow_data_sdk_version function and its call site in
tank_vendor/__init__.py (~37 lines)
- The upstream-bug commentary in the test_dist_info_via_importlib_metadata
docstring (the assertion itself is unchanged and still passes)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The required parameter was only meaningful in the late-stage-exception branch (raise vs warn). The other two failure paths returned False identically in both modes, so the parameter was mostly dead weight. Collapse to always-raise for any zip's wholesale load failure, matching the original pkgs.zip posture. Shared zips in requirements/any/ are now held to the same standard: a corrupt or import-broken shared zip will fail import tank_vendor with a clean RuntimeError instead of silently degrading. Per-package ImportError inside the loop still warns and skips, so flow_data_sdk being absent on Python 3.7/3.9 (3.10+ syntax) remains non-fatal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes a regression in the share_core integration test on Windows / Python 3.13. flow_data_sdk's _version.py calls importlib.metadata.version( "flow-data-sdk") at import time. importlib.metadata iterates sys.path in order, and FastPath.zip_children() is @lru_cache'd — the cached FastPath holds an open zipfile.ZipFile to whichever zip it probed. Previously pkgs.zip was at sys.path[0] and flow_data_sdk-beta.zip at sys.path[1], so the scan probed pkgs.zip first (no match → cached open handle), then matched in flow_data_sdk-beta.zip. The lingering handle on pkgs.zip caused share_core's shutil.move to fail with WinError 32 when relocating install/core on Windows. Reorder so shared zips end up at sys.path[0], pkgs.zip at sys.path[1]. importlib.metadata then short-circuits on the first probe and never opens pkgs.zip. pkgs.zip is still loaded into sys.modules first, so collision precedence is unchanged. Drop the path_position parameter from _load_packages_from_zip — every zip is now always inserted at sys.path[0], and the call order in tank_vendor/__init__.py determines the final sys.path ordering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… zips The previous reorder commit (f96c7d6) moved the share_core WinError 32 from pkgs.zip to flow_data_sdk-beta.zip — same root cause, different file. This is the actual fix. importlib.metadata.FastPath.__new__ is @lru_cache'd. The FastPath instance for whichever zip importlib.metadata probes is kept alive forever, and inside FastPath.zip_children() the line `self.joinpath = zip_path.joinpath` binds a zipfile.Path (with its underlying open ZipFile) as an instance attribute. The result: the cache permanently pins an open file handle on every zip that ever yielded a metadata match. flow_data_sdk's _version.py triggers this by calling importlib.metadata.version("flow-data-sdk") at module import time. The cached FastPath then keeps flow_data_sdk-beta.zip open, which on Windows blocks share_core's shutil.move(install/core, ...). Fix: after all zips are loaded, call MetadataPathFinder().invalidate_caches() to drop FastPath references, then gc.collect() so the underlying ZipFile.__del__ fires immediately and releases the OS handle. invalidate_caches is called on an instance, not the class, because it isn't decorated as @classmethod in older Python versions but takes `cls` by convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cleanup added in 8c006ca (clearing FastPath cache + gc.collect to release zipfile handles) is a workaround for Windows' sharing-violation file-move semantics. Linux and macOS allow moving files with open handles, so the cleanup is unnecessary there — and it was observed to break a Linux / Python 3.13 integration test in CI. Guard with sys.platform == "win32" so non-Windows platforms get the previous behaviour unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes prompted by PR review on #1098: 1. Broaden per-package import catch from ImportError to Exception. The inner try/except is best-effort by design (the outer wholesale-failure handler still raises). A future shared vendor using PEP 604 unions or other syntax-level newness would currently break import tank_vendor on older Pythons with a SyntaxError escaping the inner catch; widening the except matches the documented intent. 2. Add TestFlowDataSDKAbsentOnOldPython, gated to Python < 3.10. Pins the contract that the loader warns and continues when a shared vendor fails to import, so the PR's behavioural claim ("on 3.7/3.9 the SDK is simply absent") is actually exercised in CI rather than just asserted in the description. 3. Soften the misleading "mandatory" label on pkgs.zip in the module docstring. Missing pkgs.zip is tolerated to support pip-installed tk-core where dependencies come from the environment; the docstring for _load_packages_from_zip already says so, but the top-of-file summary still claimed otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Filter top-level .dll files in _discover_top_level_packages so a stray Windows DLL at the root of a vendor zip is not treated as an importable package (Carlos). - Reword the unreadable-zip warning to acknowledge that affected dependencies may still resolve from the Python environment instead of implying a guaranteed failure (Copilot). - Pass RuntimeWarning + stacklevel=2 on per-package import failures so they match the other warnings in this module and point at the caller (Copilot). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a Flow/MEDM authentication path that triggers proactively during
ToolkitManager bootstrap when the resolved project is "AM-ready" (project
field sg_flow_am_id is set). On the silent path (keyring or refresh
token), no UI surfaces; otherwise a browser PKCE flow opens before the
engine starts.
Sourced from the tk-framework-flowam PoC, which will be deleted in time.
* python/tank_vendor/adsk_auth/ vendored PKCE + keyring helper
(PyJWT + keyring as new third-party
deps, added to per-Python pkgs.zip
via requirements/<py>/requirements.txt)
* python/tank/authentication/flow_auth/
init_authentication(), get_access_token(), check_token_expiry()
FlowAuthSettings + resolve_flow_auth_settings() — defaults + env-var
overrides (TK_FLOW_AUTH_APPLICATION_ID/BASE_URL/CALLBACK_URL)
FlowAuthError / FlowAuthConfigurationError
AM_READY_PROJECT_FIELD (single point of truth — currently
sg_flow_am_id pending confirmation from Julien)
* python/tank/bootstrap/manager.py
_resolve_project_id() extracted from _get_configuration
_check_and_trigger_am_auth() new hook called right before
_get_updated_configuration returns. Configuration errors raise
TankBootstrapError; runtime auth failures are logged and swallowed
unless TK_FLOW_AUTH_REQUIRED=1.
Tests: 29 new (16 flow_auth unit + 13 bootstrap hook), all existing
bootstrap (103) and authentication (101) tests still pass.
FlowAuthenticationHandler from flowam is intentionally dropped — out
of scope for this ticket; tk-core only triggers auth here, it does not
own a GQL handler.
TODOs flagged in code for follow-up: confirm AM-ready field name and
real production APS values (application_id, base_url, callback_url) with
Julien Langlois before release. pkgs.zip regeneration is a release-time
step and not included in this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Restore the unicode → arrow in the _install_import_hook docstring. - Wrap the new requirements/any/ paragraph in developer/README.md. - Restore the Step 1/2/3/4 navigational comments inside _load_packages_from_zip that the refactor had stripped. - Drop the Python<3.8 fallback in _release_importlib_metadata_handles; Python 3.7/3.8 compatibility was discontinued after March 2026. - Reorganise __init__.py so all helper defs (including _release_importlib_metadata_handles) come before the MAIN INITIALIZATION block, and call the release helper from the main block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nto SG-43166-integrate-medm-auth
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add Sphinx-style type info to _resolve_project_id and _check_and_trigger_am_auth - Change _check_and_trigger_am_auth to accept entity instead of project_id, merging the ShotGrid AM-ready field fetch with project resolution to save one API round-trip (deep-field notation for the general entity case) - Drop the redundant _resolve_project_id call in _get_updated_configuration Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sertion Regenerated pkgs.zip, frozen_requirements.txt, and certs for Python 3.7, 3.9, 3.10, 3.11, and 3.13. Also fixed update_python_packages.py to count .dist-info directories instead of package directories when asserting install completeness, since namespace packages (ruamel.yaml, jaraco.*) share parent directories and caused a false assertion failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1. Fix publish_draft to not mutate caller's component list to avoid duplicate components on retry. 2. Include open_explorer to utils.py which is needed for publisher to resolve publish conflict.
Co-authored-by: Copilot <copilot@github.com>
Restores all files under requirements/ to their state on master. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yungsiow
commented
Jun 22, 2026
yungsiow
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1108 +/- ##
==========================================
+ Coverage 79.48% 80.28% +0.79%
==========================================
Files 206 203 -3
Lines 21009 19674 -1335
==========================================
- Hits 16700 15796 -904
+ Misses 4309 3878 -431
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced Jun 22, 2026
yungsiow
marked this pull request as ready for review
June 22, 2026 20:30
carlos-villavicencio-adsk
approved these changes
Jun 22, 2026
carlos-villavicencio-adsk
left a comment
carlos-villavicencio-adsk
left a comment
Contributor
There was a problem hiding this comment.
So far looks good. We're porting a logic that was previously reviewed anyway!
chenm1adsk
reviewed
Jun 23, 2026
chenm1adsk
left a comment
chenm1adsk
left a comment
Contributor
There was a problem hiding this comment.
The remaining looks good.
feat: [SG-43645] Migrate schema_builder module to tk-core Port SchemaBuilder and create_pipeline_schemas() from tk-framework-flowam into tk-core's flow_integration_sdk vendor package and wire schema provisioning into the Flow initialization sequence. - Add schema_builder.py to flow_integration_sdk, ported from tk-framework-flowam with updated imports and config path handling - Add FlowSchema*Error exception classes to exceptions.py - Add FLOW_SCHEMA_VERSION_FIELD constant to tank/flowam/constants.py - Move schema.get_schema_config_version() from schema_builder.py to schema.py with explicit config_path argument - Cache flow_project_id and schema_version on ToolkitManager during _get_updated_configuration() and inject onto ctx.project in _start_engine() so engine context exposes them via flow_project_id and flow_schema_version - Extend flow_utils.init_flow() to accept context object directly; add schema provisioning block with version gating and CPA check - Update Engine.__init__() call site to pass new init_flow() arguments - Fix circular import and update tests for new instance attrs and to_dict() round-trip
1 task
Merged
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Details: