Repository navigation
Open
Labels
PerformanceIssues related to fail running ABACUSIssues related to fail running ABACUS
Description
Details
Dear Developers,
I encountered an issue where using out_proj_band in large-system nscf calculations leads to very high memory consumption. For example, for a 546-atom twisted MoTe_2 (with SOC, using the TDZP+10au basis set) and a single-k point nscf calculation:
- With out_proj_band 0, I can complete the band structure calculation using 2 nodes and 96 cores total (120-core/node, mem ~500GB/node).
- With out_proj_band 1, I have to scale up to 6 nodes and 96 cores total to reduce per-rank/per-node memory pressure and complete the calculation. Otherwise, it fails with an out-of-memory error:
Info: Local MPI proc number: 48,OpenMP thread number: 1,Total thread number: 48,Local thread limit: 120
Info: Local MPI proc number: 48,OpenMP thread number: 1,Total thread number: 48,Local thread limit: 120
slurmstepd: error: Detected 1 oom_kill event in StepId=170631.0. Some of the step tasks have been OOM Killed.
srun: error: cn09: task 0: Out Of Memory
srun: Terminating StepId=170631.0
slurmstepd: error: Detected 1 oom_kill event in StepId=170631.0. Some of the step tasks have been OOM Killed.
Could you explain why this happens? Is the projected band calculation concentrated on a subset of ranks, or are there specific parameters I should configure to reduce its memory footprint? Here is my INPUT:
INPUT_PARAMETERS
#Parameters (General)
pseudo_dir /path/to/my/home
orbital_dir /path/to/my/home
#Parameters (Accuracy)
ecutwfc 100
scf_nmax 60
scf_thr 1e-7
basis_type lcao
vdw_method d3_bj
smearing_method gauss
smearing_sigma 0.01
calculation nscf
init_chg file
out_band 1
# out_proj_band 1
lspinorb 1
Task list for Issue attackers (only for developers)
- Reproduce the performance issue on a similar system or environment.
- Identify the specific section of the code causing the performance issue.
- Investigate the issue and determine the root cause.
- Research best practices and potential solutions for the identified performance issue.
- Implement the chosen solution to address the performance issue.
- Test the implemented solution to ensure it improves performance without introducing new issues.
- Optimize the solution if necessary, considering trade-offs between performance and other factors (e.g., code complexity, readability, maintainability).
- Review and incorporate any relevant feedback from users or developers.
- Merge the improved solution into the main codebase and notify the issue reporter.
Metadata
Metadata
Assignees
Labels
PerformanceIssues related to fail running ABACUSIssues related to fail running ABACUS
Activity
ABACUS-PR-BOT commentedon Nov 20, 2026
Enabling projected bands can add substantial replicated memory on every MPI rank in the LCAO/SOC output path. The projection products are distributed; rank 0 receives the summed result and writes the XML. The task-0 OOM message alone does not show that the work or large allocations are confined to rank 0.
In the inspected implementation, each rank allocates complex and real weight arrays with
K × nbands × NLOCALelements. HereNLOCALincludes both spinor components for SOC. With the usual double/complex-double sizes, these arrays occupy about24 × K × nbands × NLOCALbytes per rank. A real-valued copy raises that weight-buffer contribution to about32 × K × nbands × NLOCALbytes during reduction, in addition to distributed scratch and existing solver storage.With the same 96 ranks, spreading evenly from 2 to 6 nodes changes 48 to 16 copies per node. That reduces the node-level contribution of these replicated arrays by a factor of three, without reducing their per-rank size, and is consistent with your observation.
For this single-k-point calculation:
nbandsreduces these buffers linearly. Keep all occupied states, sufficient empty states for smearing, and the conduction-band range you need; this parameter changes the states computed, not just those written.kpardoes not partition these output buffers, andmem_saverdoes not reduce them.This is static analysis of develop commit
eba09c937c73; I have not reproduced your OOM or identified its exact failing allocation. To match the explanation to your executable, please share its version/commit, the enabled run's INPUT/INPUT.info and running_nscf.log (startup dimensions/solver and final lines), and the Slurm launch script with memory requests and rank placement for both layouts. Peak-memory accounting would help if available. The pastedout_proj_bandline is commented, so the actual enabled-run input matters.