Skip to content

out_proj_band leads to out-of-mem #8349

Open

Description

@Leiqiao-9574

Details

Dear Developers,
I encountered an issue where using out_proj_band in large-system nscf calculations leads to very high memory consumption. For example, for a 546-atom twisted MoTe_2 (with SOC, using the TDZP+10au basis set) and a single-k point nscf calculation:

  1. With out_proj_band 0, I can complete the band structure calculation using 2 nodes and 96 cores total (120-core/node, mem ~500GB/node).
  2. With out_proj_band 1, I have to scale up to 6 nodes and 96 cores total to reduce per-rank/per-node memory pressure and complete the calculation. Otherwise, it fails with an out-of-memory error:
Info: Local MPI proc number: 48,OpenMP thread number: 1,Total thread number: 48,Local thread limit: 120
Info: Local MPI proc number: 48,OpenMP thread number: 1,Total thread number: 48,Local thread limit: 120
slurmstepd: error: Detected 1 oom_kill event in StepId=170631.0. Some of the step tasks have been OOM Killed.
srun: error: cn09: task 0: Out Of Memory
srun: Terminating StepId=170631.0
slurmstepd: error: Detected 1 oom_kill event in StepId=170631.0. Some of the step tasks have been OOM Killed.

Could you explain why this happens? Is the projected band calculation concentrated on a subset of ranks, or are there specific parameters I should configure to reduce its memory footprint? Here is my INPUT:

INPUT_PARAMETERS
#Parameters	(General)
pseudo_dir           /path/to/my/home
orbital_dir            /path/to/my/home
#Parameters (Accuracy)
ecutwfc			 100
scf_nmax			60
scf_thr               1e-7

basis_type		lcao
vdw_method    d3_bj

smearing_method		gauss
smearing_sigma			0.01

calculation         nscf
init_chg                file
out_band                1
# out_proj_band          1
lspinorb              1

Task list for Issue attackers (only for developers)

  • Reproduce the performance issue on a similar system or environment.
  • Identify the specific section of the code causing the performance issue.
  • Investigate the issue and determine the root cause.
  • Research best practices and potential solutions for the identified performance issue.
  • Implement the chosen solution to address the performance issue.
  • Test the implemented solution to ensure it improves performance without introducing new issues.
  • Optimize the solution if necessary, considering trade-offs between performance and other factors (e.g., code complexity, readability, maintainability).
  • Review and incorporate any relevant feedback from users or developers.
  • Merge the improved solution into the main codebase and notify the issue reporter.

Activity

  1. ABACUS-PR-BOT commented on Nov 20, 2026

    @ABACUS-PR-BOT
    Collaborator

    Enabling projected bands can add substantial replicated memory on every MPI rank in the LCAO/SOC output path. The projection products are distributed; rank 0 receives the summed result and writes the XML. The task-0 OOM message alone does not show that the work or large allocations are confined to rank 0.

    In the inspected implementation, each rank allocates complex and real weight arrays with K × nbands × NLOCAL elements. Here NLOCAL includes both spinor components for SOC. With the usual double/complex-double sizes, these arrays occupy about 24 × K × nbands × NLOCAL bytes per rank. A real-valued copy raises that weight-buffer contribution to about 32 × K × nbands × NLOCAL bytes during reduction, in addition to distributed scratch and existing solver storage.

    With the same 96 ranks, spreading evenly from 2 to 6 nodes changes 48 to 16 copies per node. That reduces the node-level contribution of these replicated arrays by a factor of three, without reducing their per-rank size, and is consistent with your observation.

    For this single-k-point calculation:

    • Fewer ranks per node/more allocated memory per rank directly helps node capacity. Check the Slurm memory allocation as well as physical node RAM.
    • If you are computing unnecessary empty bands, lowering nbands reduces these buffers linearly. Keep all occupied states, sufficient empty states for smearing, and the conduction-band range you need; this parameter changes the states computed, not just those written.
    • The inspected code has no projected-band chunk-size control. LCAO kpar does not partition these output buffers, and mem_saver does not reduce them.

    This is static analysis of develop commit eba09c937c73; I have not reproduced your OOM or identified its exact failing allocation. To match the explanation to your executable, please share its version/commit, the enabled run's INPUT/INPUT.info and running_nscf.log (startup dimensions/solver and final lines), and the Slurm launch script with memory requests and rank placement for both layouts. Peak-memory accounting would help if available. The pasted out_proj_band line is commented, so the actual enabled-run input matters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PerformanceIssues related to fail running ABACUS

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions