Add handling for global parameters - #15
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end support for Azure Data Factory (ADF) factory global parameters in flowx, letting users choose whether global references are baked in as literals or hoisted to Databricks Asset Bundle (DAB) variables that can be overridden at deploy time.
Changes:
- Added a
global_parameter_resolutionpolicy (literalvsbundle_variable) that affects expression resolution, IR rewriting, bundling, and setup guidance. - Extended ARM template ingestion to load
/globalparametersresources and resolveparameters('X')references from the sibling parameters file. - Threaded hoisted globals through the workflow/bundler so
databricks.ymldeclares variables andSETUP.mddocuments overrides (including plaintext-secret caveats), with unit tests covering the behavior.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/flowx/translator/engine.py | Adds the global-parameter resolution policy, passes it through rewrite passes, and hoists referenced globals into Pipeline.bundle_variables. |
| src/flowx/parser/expression_parser.py | Updates global parameter expression resolution and extends notebook interpolation lowering to bridge ${var.X} via widgets. |
| src/flowx/parser/ir_rewriter.py | Ensures whole-IR @{...} rewriting has access to factory globals and the resolution policy. |
| src/flowx/parser/adf_loader.py | Adds ARM parameters-file discovery + substitution and loads standalone /globalparameters resources. |
| src/flowx/models/ir.py | Adds Pipeline.bundle_variables and threads TranslationContext.global_parameter_resolution through context reconstruction. |
| src/flowx/translator/activity_translators/for_each.py | Threads the policy through ForEach child contexts. |
| src/flowx/preparer/workflow_preparer.py | Copies Pipeline.bundle_variables onto PreparedWorkflow for downstream bundling. |
| src/flowx/bundler/dab_writer.py | Declares hoisted globals in databricks.yml and binds hoisted widget defaults to ${var.X} during base-parameter augmentation. |
| src/flowx/bundler/prereqs_writer.py | Adds a SETUP.md section listing hoisted globals + deploy-time override commands and a security note. |
| src/flowx/adapter/constants.py | Adds global_parameter_resolution input constant. |
| src/flowx/adapter/session.py | Adds a convert-phase input option prompting users for global parameter resolution mode. |
| skills/flowx-convert/SKILL.md | Documents the new --global-parameter-resolution CLI flag and behavior. |
| docs/content/docs/options.mdx | Documents the new global_parameter_resolution option and its security implications. |
| AGENTS.md | Adds repo guidance for naming/docstrings/comments conventions. |
| tests/unit/test_translators.py | Adds engine-level tests asserting literal vs hoisted-global behavior, including ForEach and whole-IR rewrite cases. |
| tests/unit/test_expression_parser.py | Adds expression parser tests for bundle-variable behavior and policy propagation through context rebuilds. |
| tests/unit/test_bundler.py | Adds tests asserting hoisted globals are declared in databricks.yml, show up in SETUP.md, and bind widgets correctly. |
| tests/unit/test_adf_loader.py | Adds tests for ARM template globalparameters ingestion and parameters-file resolution helpers. |
| tests/unit/test_adapter.py | Updates adapter tests to assert the new option is present and defaults to literal. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
271
to
273
| if result.kind == "literal": | ||
| return result.value | ||
| if result.kind == "dab_ref": |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add handler methods to resolve global parameters from ADF resources.
Parameter is controlled by the user via prompting. The choice is applied to an entire ADF resource during conversion. Users can choose to:
Linked issues
N/A
Tests