My Community

I am working with a 245 atom system with C_1h symmetry. There are a total of 1372 electrons. I initially relaxed the system using the following INCAR file without trouble with 24 cores. As I recall the actual memory usage was about 64 GB for all processes (only Centos 6 was running besides Vasp) out of 128 GB of total memory. The program was run on a single node. There were 4 K-points. As the structure was thin in one direction and quite large in another two (3x16x41) I used a Gamma centered 6x1x1 M-P k-point mesh. As there was some charge sloshing due to the shape, I changed some cards relating to charge mixing (AMIN), and had no problem with the SCF loop. Vasp stopped after 50 NSW cycles and the maximum force was about 1E-3. At that point I wanted to see what the band structure looked like.

INCAR FILE For relaxation

ENCUT = 250.000000
AMIN = 0.010000
SIGMA = 0.050000
EDIFF = 1.00e-05
PREC = Normal
ALGO = fast
NBANDS = 840
ISMEAR = -2
NSW = 50
IBRION = 1
ISIF = 2
LWAVE = .FALSE.
LPLANE = .TRUE.


OUTCAR (memory related fragment)

General timing and accounting informations for this job:
========================================================

Total CPU time used (sec): 835438.312
User time (sec): 828826.920
System time (sec): 6611.384
Elapsed time (sec): 838352.570

Maximum memory used (kb): 1366740.
Average memory used (kb): 0.

Minor page faults: 7489839
Major page faults: 0
Voluntary context switches: 499047415



I then attempted to run a non-self consistent band structure calculation using the CHGCAR file from the relaxation. Much to my surprise, VASP proceeded in the setup phase to use more than 128 GB of memory. I then tried the same calculation with two 24 core nodes (connected by infiniband) and Vasp proceeded to use more than 256 GB of memory (128 GB x 2). I have no idea why Vasp is trying to use so much memory and am looking for suggestions. I should add that the memory problem persisted even when I changed the number of nodes. What is the cause of this inconsistent use of memory and how can I work around it.

Thanks for any helpful suggestions in advance.


KPOINTS file
50
Line-mode
Reciprocal
0.0 0.5 0.0
0.0 0.0 0.0
0.0 0.0 0.0
0.33333 0.33333 0.333333


INCAR file for band structure
AMIN = 0.010000
EDIFF = 1.00e-07
PREC = Accurate
NBANDS = 1400
ISTART = 0
NELM = 200
ICHARG = 11
LPLANE = True
IALGO = 48
XC = PBE
NGX = 19
NGY = 136
NGZ = 341
SIGMA = 0.1



OUTCAR file fragment (right before abort)

k-point 99 : 0.3265 0.3265 0.3265 plane waves: 56265
k-point ** : 0.3333 0.3333 0.3333 plane waves: 56242

maximum and minimum number of plane-waves per node : 56708 56242

maximum number of plane-waves: 56708
maximum index in each direction:
IXMAX= 4 IYMAX= 33 IZMAX= 85
IXMIN= -4 IYMIN= -34 IZMIN= -85

NGX is ok and might be reduce to 18
NGY is ok and might be reduce to 136
NGZ is ok and might be reduce to 342

serial 3D FFT for wavefunctions
parallel 3D FFT for charge:
minimum data exchange during FFTs selected (reduces bandwidth)

(I aborted at this point as memory use was more than 128 GB)

as far as I see, the number of k-points was increased from 4 to 100 , this
is the reason for having not enough memory in the last run.