Skip to content

Distribute DeePMD inference through MDCell for UnitCell workflows - #8330

Merged
mohanchen merged 15 commits into
deepmodeling:developfrom
19hello:fix/dp-mdcell-8310
Nov 20, 2026
Merged

mohanchen merged 15 commits into
deepmodeling:developfrom
19hello:fix/dp-mdcell-8310

Conversation

@19hello

@19hello 19hello commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue.
  • I have added focused unit tests and recorded local model runs.
  • I have listed verification commands, results, and limitations.
  • I have described user-visible behavior and INPUT impact.
  • I have explained the affected core modules.
  • No governance exception is requested.

Linked Issue

Refs #8310. The original 8192-atom run on two A800 80 GB GPUs with DeePMD-kit 3.1.1 remains to be verified before claiming the issue is fully fixed.

What's changed?

  • Add a source_cell MDCell adapter that initializes a domain-decomposed MDCell from STRU or UnitCell, synchronizes coordinates and lattice geometry, returns ghost forces to owners, and gathers a full force matrix for UnitCell optimizers. The driver passes concrete INPUT values to the adapter; the adapter handles restart replication, skin conversion, world-domain construction, atom-count logging, and the UnitCell backing association. It does not receive Parameter or use global PARAM. Run_MD::prepare_mdcell and its MDCell friend declaration were removed.
  • Route both DP cell kinds through one local-neighbor-list inference function. The UnitCell path keeps its distributed MDCell across calculations; relax and cell-relax keep their existing UnitCell-facing optimizer interface. Energy and virial use the MDCell cross-rank reductions.
  • Support dim_aparam > 0 by mapping either one parameter vector or global nat * dim_aparam values to owned atoms and, when the model requires it, ghosts. Keep fparam frame-scoped.
  • Select the DeePMD GPU rank from the node-local MPI rank or ABACUS GPU binding; probe the usable DeePMD C++ interface and communication-capable InputNlist; reject multi-rank inference when the latter is unavailable. Handle ranks with no local atoms.
  • Document distributed DP behavior, atomic-parameter order, and GPU-sharing limits in docs/advanced/md.md.

Unit Tests and/or Case Tests for my changes

  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_CELL_mdcell_adapter MODULE_MD_dp_aparam -j8 — passed with local DeePMD 3.2.1.dev26 PyTorch build.
  • ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^(MODULE_CELL_mdcell_adapter(_np[24])?|MODULE_MD_dp_aparam|MODULE_MD_run)$' — 5/5 passed outside the restricted sandbox.
  • Non-MPI build of md and domain_decomposition — passed. DeePMD 2.2.11 C-API fallback md compile — passed.
  • With the reporter-provided DPA model: 512-atom Fe, two MPI ranks, one GPU — completed; 128-atom two-rank relax and cell-relax each completed two steps. A sparse two-atom case gave identical energy, forces, and pressure at one and four ranks, including completely empty ranks.
  • With a locally generated dim_aparam=1 DeePMD model: per-atom values 0.1 0.2 and uniform value 0.1 each gave matching one- and two-rank results; changing the parameters changed the energy.
  • python3 tools/03_code_analysis/agent_governance_check.py --base origin/develop --head HEAD --format text — no blockers, only four header-include review warnings. git diff --check origin/develop..HEAD — passed. Code-quality score: 8/8 changed C++ files passed.

Checks not completed or passed:

  • The original 8192-atom/two-A800/DeePMD 3.1.1 configuration is unavailable locally. This workstation has one 11 GB RTX 2080 Ti. At 1024 atoms, two ranks sharing that GPU exhausted its memory; this does not establish the memory limit when ranks use separate A800s.

  • Existing MODULE_MD_DP_potential uses Al-SCAN.pb and failed because this local DeePMD installation lacks libdeepmd_backend_tf.so. The older 2.2.11 installation also lacks required TensorFlow link dependencies. Neither result was counted as a pass.

  • The reporter model and the locally generated atomic-parameter model are not committed to the repository. A committed manual rank-comparison script and fixture now reproduce the reporter-model check; no model-backed Autotest case is registered because the model is an issue attachment.

  • Follow-up initialization refactor: source /home/fy19/abacus-develop/toolchain/install/setup && cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_MD_run MODULE_CELL_mdcell_adapter -j4 — passed. OMP_NUM_THREADS=1 ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R 'MODULE_MD_run|MODULE_CELL_mdcell_adapter' — 4/4 passed (adapter at 1, 2, and 4 ranks, plus Run_MD).

  • After moving setup decisions into the adapter, the executable and MODULE_CELL_mdcell_adapter rebuilt successfully. The adapter regression test checks normal replication (2 atoms), restart without repeated replication (1 atom), and retention of the atom-count log; OMP_NUM_THREADS=1 ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R 'MODULE_MD_run|MODULE_CELL_mdcell_adapter' passed 4/4, including 1/2/4-rank adapter runs.

  • Temporary runtime cases with this build: two-rank LJ Anderson MD with cell_replica 2 1 1 and md_neighbor_skin 1.0 (64 atoms), PW NVT restart (2 atoms), and two-rank LJ NVE restart with cell_replica 2 1 1 (checkpoint remained 32 atoms) — all exited successfully; the atom-count log remained present. These smoke cases were run from /tmp, not committed.

  • python3 tools/03_code_analysis/agent_governance_check.py --staged and git diff --cached --check — passed with no blockers. The final adapter revision has no blockers: the checker reported a reduced global-reference count (1 added, 3 removed), a review warning for <iosfwd> required by the std::ofstream& declaration, and a no-docs-needed warning. This follow-up changes no INPUT keyword, log format, or public user behavior.

Governance Notes

  • INPUT/docs changes: no keyword or parser changes. dp_aparam now works in the MDCell inference path with its existing documented input shapes; docs/advanced/md.md describes the parallel behavior.
  • Core module impact: source_cell owns the adapter and empty-domain neighbor handling; source_md DP uses it. The external ESolver and UnitCell optimizer interfaces are unchanged. New source files are listed in CMake and source/Makefile.Objects.
  • Header dependency warnings: the new adapter and atomic-parameter headers need std::string, std::vector, and std::int64_t in their public declarations; concrete cell types are forward-declared.
  • Exceptions requested: none.

Latest follow-up verification

  • a04ba4094 unified MDCell lattice updates through set_lattice_geometry(latvec, lat0); NHC and MSST pass the existing lat0. The MPI decomposition, adapter, NHC, MSST, and LJ tests passed (6 tests total) after the affected targets built.

  • e442366b6 inlined the output/restart cadence checks without changing their predicates. source /home/fy19/abacus-develop/toolchain/install/setup && cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_MD_func -j 8 passed; ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^MODULE_MD_func$' passed. A seven-step LJ MD smoke run with out_freq_ion=3 and md_restartfreq=5 produced STRU1, STRU6 for out_stru=0, and STRU1, STRU4, STRU6, STRU7 for out_stru=1; both wrote restart state at step 6. The earlier predicate-only MODULE_MD_run test target was removed with those predicates. An initial CTest command also selected three targets not built in the DPA build directory, so they were Not Run; this was a build-selection issue, not a test failure.

  • python3 tools/03_code_analysis/agent_governance_check.py --staged --format text passed with only a documentation-sync review warning. git diff --cached --check passed. These follow-up changes do not alter INPUT behavior or user-facing documentation.

  • e0c6d61a1 shortened only comments in run_md.cpp (11 comment lines added, 60 removed); executable statements are unchanged. git diff --cached --check and the staged governance check passed without blockers. No new runtime test or documentation is needed for this comment-only follow-up; the earlier MD runtime checks remain the behavior evidence.

  • 50d03d908 removes redundant int to std::size_t conversions only from standard-container subscripts in the DP inference and MDCell adapter paths. Bounds checks, size comparisons, and non-index conversions remain. source /home/fy19/abacus-develop/toolchain/install/setup && cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_CELL_mdcell_adapter MODULE_MD_dp_aparam -j 8 passed; ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^(MODULE_CELL_mdcell_adapter(_np[24])?|MODULE_MD_dp_aparam)$' passed 4/4. The staged governance checker had no blockers, and git diff --cached --check passed. This is an indexing-only cleanup with no INPUT or numerical behavior change, so no new docs or tests are needed.

Conflict resolution and review follow-up (2026-11-19)

  • Merged upstream develop (d75733074), resolving the forcefield source rename and migrated unit-test paths. GitHub reports the PR mergeable against current develop.
  • d11038e21 verifies that libdeepmd_op_pt.so directly depends on libmpi.so before enabling distributed DeePMD inference in both CMake and Make builds. A non-MPI operator configuration now warns and excludes the communication-capable compile definition; runtime rejects multi-rank inference. The MD guide requires the operator and ABACUS to use the same MPI runtime. The resolved MPI library was checked locally with ldd.
  • Added a dedicated LJ output-cadence CTest check for actual STRU frame inventory and restart output with out_stru=0/1; exact file inventory cannot be expressed by the numerical Autotest reference checks. Added a manual model-backed rank-comparison script and fixture, with reproduction instructions in tests/04_FF/22_DP_rank_consistency/README.md. It checks energy, all forces and stress after coordinate and lattice changes, including empty rank partitions. The model binary remains the issue DPA3 relax runs out of GPU memory with two GPUs allocated #8310 attachment.
  • Reused a checked component_count for adapter force allocation and reduction.

Fresh verification after review changes:

  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target abacus_pw_para MODULE_CELL_mdcell_adapter MODULE_MD_dp_aparam -j 8 passed with local DeePMD 3.2 development build.
  • ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^(04_FF_md_output_cadence|MODULE_CELL_mdcell_adapter(_np[24])?|MODULE_MD_dp_aparam)$' passed 5/5.
  • python3 tests/integrate/validation_tools/check_dp_rank_consistency.py --abacus /tmp/abacus-dp-mdcell-build-dpa/abacus_pw_para --mpi /home/fy19/abacus-develop/toolchain/install/openmpi-5.0.10/bin/mpirun --model /tmp/issue8310_repro/frozen_model_op.pth --fixture tests/04_FF/22_DP_rank_consistency/STRU passed for original, displaced and expanded 2-atom cells at 1 and 4 ranks. The displaced cell produced a nonzero force (max 0.7566 eV/Angstrom). Comparison tolerances match the integration-suite values: 1e-5 eV, 1e-4 eV/Angstrom and 1e-3 kbar. The model run required execution outside the restricted sandbox because OpenMPI could not open its interfaces inside it.
  • Configuring against a synthetic operator with no libmpi.so dependency emitted the expected distributed-inference-disabled warning, and the resulting flags omitted __DPMD_COMM_NLIST_ARGS; the real MPI-enabled operator retained __DPMD_COMM_NLIST_ARGS=13.
  • python3 tools/03_code_analysis/agent_governance_check.py --staged reported no findings; git diff --cached --check passed. The full PR diff checker reports only review warnings for migration-neutral global use and header includes, with no blockers.

CI build fix (2026-11-19)

  • The GNU integration/unit-test and CUDA jobs failed during compilation of the migrated test_domain_decomposition.cpp: two tests still called removed MDCell::set_lattice_vectors. Commit 7fd9105b2 updates those calls to set_lattice_geometry(lattice, lat0) while preserving the original lattice constant. This is a test-only call-site correction; no user-facing behavior or documentation changes are needed.
  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target MODULE_CELL_NEIGHBOR_mdcell_migrate_mpi -j 8 passed.
  • ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^MODULE_CELL_NEIGHBOR_(mdcell_migrate_mpi|domain_decomposition_np[14])$' passed 3/3 at 2, 1, and 4 MPI ranks.
  • python3 tools/03_code_analysis/agent_governance_check.py --staged had only the expected documentation-sync review warning for a test-only change; git diff --cached --check passed.

Review follow-up and documentation check (2026-11-19)

  • b75ca502f restores mdcell_adapter.cpp and mdcell_reader.cpp to MODULE_MD_dp_forcefield's serial test sources; preserves ordinary distributed TensorFlow .pb inference when the PyTorch MPI operator is unavailable; checks the LJ restart's actual step index; and extends the model comparison to two-step relax and cell-relax on persistent DP/MDCell instances. The MD guide now distinguishes PyTorch and TensorFlow requirements.

  • cmake --build /tmp/abacus-dp-mdcell-build-dpa --target MODULE_MD_dp_forcefield abacus_pw_para -j 8 passed with DeePMD enabled. ctest --test-dir /tmp/abacus-dp-mdcell-build-dpa --output-on-failure -R '^04_FF_md_output_cadence$' passed 1/1.

  • cmake --build /tmp/abacus-dp-mdcell-build-no-dp --target MODULE_MD_dp_forcefield -j 8 passed without DeePMD; its CTest passed 1/1. The same target compiled and linked against a synthetic DeePMD installation without an MPI-enabled PyTorch operator, exercising the compile-time guard branch.

  • python3 tests/integrate/validation_tools/check_dp_rank_consistency.py --abacus /tmp/abacus-dp-mdcell-build-dpa/abacus_pw_para --mpi /home/fy19/abacus-develop/toolchain/install/openmpi-5.0.10/bin/mpirun --model /tmp/issue8310_repro/frozen_model_op.pth --fixture tests/04_FF/22_DP_rank_consistency/STRU passed all three SCF geometries plus two consecutive evaluations each for relax and cell-relax at one and four ranks. A TensorFlow .pb runtime check was unavailable because the local DeePMD package has no TensorFlow backend library.

  • python3 tools/03_code_analysis/agent_governance_check.py --staged and git diff --cached --check passed with no findings.

  • The docs/readthedocs.org:abacus-rtd check on 7fd9105b2 failed at the 900-second platform limit before any project documentation build command ran. Its public build record stopped after apt-get update and before apt-get install began; the same check had passed on d11038e21. This is recorded as a documentation service timeout, not a Sphinx diagnostic. The new commit retriggers it.

  • On b75ca502f, both Read the Docs contexts (abacus-develop and abacus-rtd) completed successfully, confirming the previous 900-second abacus-rtd failure was transient. The four review threads addressed by this follow-up were replied to and resolved.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source inspection identifies a conditional MPI-backend compatibility risk in the new UnitCell inference route. The inline comments also include three non-blocking follow-ups. This was a static review; no builds or runtime tests were run.

Comment thread source/source_md/run_md.cpp Resolved
Comment thread source/source_cell/mdcell_adapter.cpp Resolved
@mohanchen mohanchen added MD & LAM MD and Larege Atomic Models Refactor Refactor ABACUS codes labels Nov 19, 2026

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static follow-up of 7fd9105b243f; no builds or runtime tests were run. The checked component_count is now reused for allocation and reduction.

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental static review of b75ca502f260; builds and tests were not run.

Comment thread source/source_cell/mdcell_adapter.cpp Resolved
@mohanchen
mohanchen merged commit 672e0fa into deepmodeling:develop Nov 20, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

MD & LAM MD and Larege Atomic Models Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants