Skip to content

Support immutable arrays in flat correction and masking - #956

Merged
mwcraig merged 2 commits into
astropy:mainfrom
nomad3:fix-931-immutable-array-updates
Jul 29, 2026
Merged

Support immutable arrays in flat correction and masking#956
mwcraig merged 2 commits into
astropy:mainfrom
nomad3:fix-931-immutable-array-updates

Conversation

@nomad3

@nomad3 nomad3 commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • rebind functional xpx.at(...).set(...) results in flat_correct and ccdmask so immutable Array API backends retain the updates
  • use backend-portable stacking for column statistics and flatten blocks with reshape((-1,)) so JAX/Dask reach the update paths
  • add deterministic, backend-native regressions for masked flats, minimum flat values, block masks, all-masked columns, and column-gap filling while checking that inputs remain unchanged

Testing

  • pytest -q ccdproc: 350 passed, 29 skipped
  • CCDPROC_ARRAY_LIBRARY=jax JAX_ENABLE_X64=True pytest -q ccdproc: 343 passed, 29 skipped, 7 expected failures
  • CCDPROC_ARRAY_LIBRARY=dask pytest -q ccdproc: 345 passed, 34 skipped
  • focused Array API strict run: 4 passed, 1 expected failure at the pre-existing CCDData mask-copy boundary
  • Black, Ruff, and git diff --check

Fixes #931

Checklist

  • For new contributors: Did you add yourself to the AUTHORS.rst file? (Present on current main through merged Exclude masked weights from average combinations #952.)
  • For documentation changes: Does your commit message include a [skip ci]? (Not a documentation-only change.)
  • For bugfixes: Did you add an entry to the CHANGES.rst file?
  • For bugfixes: Did you add a regression test?
  • For bugfixes: Does the commit message include Fixes #931?
  • Does this PR add, rename, move or remove any existing functions or parameters?

AI assistance disclosure

OpenAI Codex was used for source inspection, test design, implementation, and automated review. The contributor has completed the final review and takes responsibility for the contribution's correctness and maintenance.

nomad3 added a commit to nomad3/ccdproc that referenced this pull request Jul 19, 2026
Signed-off-by: Simon Aguilera <saguilera1608@gmail.com>
Use functional updates in flat_correct and ccdmask, and add backend-native regression coverage for masked flats, block masks, and column-gap filling.

Fixes astropy#931

Signed-off-by: Simon Aguilera <saguilera1608@gmail.com>
@nomad3
nomad3 force-pushed the fix-931-immutable-array-updates branch from 73009bb to c26251b Compare July 27, 2026 04:15
@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.70%. Comparing base (e2d43f3) to head (3ea1af9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #956      +/-   ##
==========================================
+ Coverage   96.63%   96.70%   +0.06%     
==========================================
  Files           8        8              
  Lines        1575     1576       +1     
==========================================
+ Hits         1522     1524       +2     
+ Misses         53       52       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nomad3
nomad3 marked this pull request as ready for review July 27, 2026 13:00
@mwcraig
mwcraig requested a review from Copilot July 27, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates flat_correct and ccdmask to avoid in-place item assignment so the routines work correctly with immutable Array API backends (e.g., JAX), and adds regression tests to ensure both correctness and input immutability across supported backends.

Changes:

  • Rebind xpx.at(...).set(...) results in flat_correct and ccdmask so updates are preserved on immutable arrays.
  • Replace backend-fragile flattening/stacking with portable reshape((-1,)) and xp.stack(...) to exercise the intended code paths on JAX/Dask.
  • Add backend-native regression tests covering masked flats, min_value behavior, block masking, and column-gap filling, including checks that inputs remain unchanged.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
CHANGES.rst Adds a changelog entry documenting the functional-update fix for immutable Array API backends.
ccdproc/core.py Replaces in-place item assignment with functional array updates in flat_correct and ccdmask and improves backend portability.
ccdproc/tests/test_ccdproc.py Adds/updates regressions for flat_correct to validate masked-flat behavior and immutability on Array API backends.
ccdproc/tests/test_ccdmask.py Adds regressions ensuring ccdmask update paths work with immutable backends and that inputs are not modified.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mwcraig mwcraig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling #931 — the production changes here are exactly right. Independent verification: numpy results are bit-identical before/after this PR across a broad sweep of ccdmask/flat_correct configurations, jax now matches numpy, and on main+jax flat_correct(min_value=...) was silently ignoring min_value (the xpx.at result was discarded), so this fixes silent wrong results as well as crashes. The new tests genuinely fail against unfixed code on jax, so the regression value is real.

I am requesting one round of changes, confined to the new ccdmask tests: the findbadcolumns=True case executes the ported column-statistics code, but its assertions cannot detect a value-level regression in it. Concretely (verified by mutation testing): changing the all-masked fill to .set(99) and reintroducing the discarded-result form xpx.at(csum)[csum <= 0].set(0) — the exact bug class this PR fixes — leaves all three new tests passing on both numpy and jax. The cause is the test data: with the mocked filters, the unmasked residuals are all zero, so csum == [0, 0, 0, 0] in every block, the set(0) fill maps zeros to zeros, and colmask needs |csum| > 3 * csum_sigma ≈ 6 to fire, which never happens. The tell: the expected mask is identical for findbadcolumns=False and =True.

Disclosure: these review comments were generated by Claude Code at my suggestion — including the code trace and the mutation testing described above — and reviewed/edited by me before posting.

Comment thread ccdproc/tests/test_ccdmask.py Outdated
Comment thread ccdproc/tests/test_ccdmask.py
Comment thread ccdproc/tests/test_ccdmask.py
Signed-off-by: Simon Aguilera <saguilera1608@gmail.com>

@mwcraig mwcraig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again, @nomad3!

@mwcraig
mwcraig merged commit 6a0240c into astropy:main Jul 29, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Array API: in-place item assignment in flat_correct and ccdmask breaks immutable backends

3 participants