Skip to content

feat: Add optional static_url to BaseOperatorLink to skip XCom for constant links - #70343

Open
iRAFEEK wants to merge 7 commits into
apache:mainfrom
iRAFEEK:feat/static-operator-link
Open

feat: Add optional static_url to BaseOperatorLink to skip XCom for constant links#70343
iRAFEEK wants to merge 7 commits into
apache:mainfrom
iRAFEEK:feat/static-operator-link

Conversation

@iRAFEEK

@iRAFEEK iRAFEEK commented Jul 23, 2026

Copy link
Copy Markdown

Closes #55432

Problem

BaseOperatorLink.get_link() was an @abstractmethod, forcing every
subclass to implement it even when the link URL is a constant (e.g.,
a documentation link). This meant operators with static links still had
to push to XCom during execute() and read from the metadata database
on every task run, adding unnecessary overhead.

Solution

This PR introduces an optional static_url property on
BaseOperatorLink:

  • static_url defaults to None (no behavior change for existing subclasses)
  • get_link() is no longer @abstractmethod; its default implementation
    returns static_url if set, otherwise raises NotImplementedError
    with a helpful message guiding the developer to override one or the other
  • Subclasses with a constant URL can now override static_url instead
    of get_link(), with zero XCom involvement

Backward Compatibility

Fully backward-compatible. All existing subclasses that implement
get_link() continue to work without any changes.

Example

class MyDocsLink(BaseOperatorLink):
    name = "Documentation"
    operators = [MyOperator]

    @property
    def static_url(self) -> str:
        return "https://myoperator.readthedocs.io"
    # No get_link() needed — no XCom push during execute()

Tests

Added task-sdk/tests/task_sdk/bases/test_operatorlink.py covering:

  • static_url defaults to None
  • Subclass using static_url returns correct URL from get_link()
  • Subclass overriding get_link() directly still works (existing behavior)
  • Subclass implementing neither raises NotImplementedError with a clear message
  • name remains abstract

@boring-cyborg

boring-cyborg Bot commented Jul 23, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the PR! I wonder whether this would still write to XCom when static_url is set.

link, xcom_key = oe.get_link(operator=task, ti_key=ti), oe.xcom_key # type: ignore[arg-type]
log.debug("Setting xcom for operator extra link", link=link, xcom_key=xcom_key)
_xcom_push_to_db(ti, key=xcom_key, value=link)

It seems _xcom_push_to_db() is executed unconditionally after get_link(). Would you mind taking a look?

or push anything to XCom during task execution.

:return: A static URL string, or ``None`` to use the XCom-based :meth:`get_link`.
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be a good idea to add this informatio to the relevant site doc too.

finalize() previously called _xcom_push_to_db() unconditionally for every
operator extra link, so a link declared via static_url was still written to
the metadata DB on each task run -- even though that value is never read back
(the render path in SerializedBaseOperator.get_extra_links resolves the URL by
calling get_link() directly). This defeated the point of static_url.

Skip the XCom write when a link is static, add a task_runner test asserting no
push happens for a static link (dynamic links unchanged), and document
static_url in the define-extra-link howto.

Addresses review feedback from @aaron-y-chen and @zach-overflow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: iRAFEEK <rafeekmagdy28@gmail.com>
@iRAFEEK

iRAFEEK commented Aug 6, 2026

Copy link
Copy Markdown
Author

@aaron-y-chen good catch, you're right. finalize() was pushing to XCom unconditionally after get_link(), so a static_url link was still persisted every run. Fixed in 8349e44: skip the push when the link is static. Safe because the render path (SerializedBaseOperator.get_extra_links) calls get_link() directly and never reads that XCom back. Dynamic links are unchanged; added a test covering both.

@iRAFEEK

iRAFEEK commented Aug 6, 2026

Copy link
Copy Markdown
Author

@zach-overflow good call, added a static_url note with an example to airflow-core/docs/howto/define-extra-link.rst in the same commit (8349e44). Thanks!

iRAFEEK and others added 2 commits August 11, 2026 19:09
A newer ruff (0.16.0, pulled in via main) flags PT012 on the
pytest.raises block in test_operatorlink.py (it contained a class
definition plus the instantiation). Move the class definition out of the
with-block so only the raising call remains inside; behaviour is
unchanged (the TypeError is raised at instantiation). Also add missing
trailing newlines to operatorlink.py and test_operatorlink.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: iRAFEEK <rafeekmagdy28@gmail.com>
@iRAFEEK

iRAFEEK commented Aug 13, 2026

Copy link
Copy Markdown
Author

Thanks again for the reviews @aaron-y-chen @zach-overflow! I've pushed:

  • the XCom fix (skip the _xcom_push_to_db() call in finalize() for static links, since the render path resolves them via get_link() directly) + a test, and
  • the docs note for static_url, and
  • a small lint follow-up (ruff PT012 + trailing newlines) after the newer ruff landed via main.

Everything's green locally (including ruff==0.16.0). Could a committer kick off / approve the CI workflows when you have a moment? Happy to address anything else. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optional support for static Extra Links without XCom dependency

3 participants