Since MPI jobs use memory, consider memory allocation as shown in the following examples.
S queue is mounting 54GB memory but if you use 50GB or less in consideration of the amount of memory the system uses, it will work efficiently.
However, when you use MPI which needs some amount of memory for high-speed processing, allow a margin of memory like followings.
The memory which the process you run requires is assumed to be 3 GB.
Since the processes use around 1GB memory when the number of processes to communicate with is tens, your process needs 4GB memory to run.
In addition, in order to run MPI, buffered memory for the system is reserved independently.
It is said that the memory size at that time depends on its behavior and can be 5GB at maximum.
So safe operation will be ensured if the memory used by a chunk is 45 GB or less
("chunk" is the part of ":ncpus=4:mpiprocs=4:mem=16gb" described below "select=x:").
In accordance with the example above, the memory used within a node is 4 GB per process.
If the total amount of memory is 45 GB or less you can run up to 11 processes on a node and it needs two nodes.
In this case, since the process becomes stable if use of nodes is equalized,
configure the resource settings like this when submitting jobs.
-l select=2:ncpus=8:mpiprocs=8:mem=32gb -l place=scatter
(See also Chapter 5.)
Like this example, think about resource settings in consideration of the balance of the memory and the number of processes.