This shows you the differences between two versions of the page.
Next revision | Previous revision | ||
pinnacle_usage [2020/09/21 22:00] root created |
pinnacle_usage [2022/01/27 17:13] root |
||
---|---|---|---|
Line 4: | Line 4: | ||
Pinnacle has 101 compute nodes. 30 GPU and GPU-ready nodes are Dell R740, 69 nodes are Dell R640, two nodes are Dell R7425. There is no user-side difference between R740 (GPU-ready) and R640 nodes. | Pinnacle has 101 compute nodes. 30 GPU and GPU-ready nodes are Dell R740, 69 nodes are Dell R640, two nodes are Dell R7425. There is no user-side difference between R740 (GPU-ready) and R640 nodes. | ||
- | All-user nodes number 76, of which 6 nodes have 768 GB of memory and no GPU, 19 nodes have 192 GB and one V100 GPU, and 51 are standard compute nodes with 192 GB and no GPU. | + | All-user nodes number 76, of which 6 nodes have 768 GB of memory and no GPU (himem.. partition), 19 nodes have 192 GB and one V100 GPU (gpu.. partition), and 51 are standard compute nodes with 192 GB and no GPU (comp.. partition). |
- | Condo nodes number 25, including 20 Wang, standard compute nodes with NVMe drives, two Alverson, one 192 GB standard and one 768 GB, one Kaman, 768 GB with two V100 GPU, and two Barry, 64-core AMD with 256 GB. | + | Standard nodes have two Gold 6130 CPUs with total 32 cores at 2.1 GHz. 768 GB nodes have two Gold 6126 CPUs with total 24 cores at 2.6 GHz, fewer and faster cores for better performance on often poorly-threaded bioinformatics applications. |
- | + | ||
- | Standard nodes have two Gold 6130 CPUs with total 32 cores at 2.1 GHz. 768 GB nodes have two Gold 6126 CPUs with total 24 cores at 2.6 GHz, fewer and faster cores for better performance on often poorly-threaded bioinformatics applications. The two R7425 nodes have dual Epyc 7351 processors, total 32 cores at 2.4 GHz. | + | |
===Login=== | ===Login=== | ||
Line 14: | Line 12: | ||
===Scheduler=== | ===Scheduler=== | ||
- | We are transitioning to Centos 7 and the SLURM scheduler | + | All systems now use the slurm scheduler.Queues (slurm " |
- | + | ||
- | Queues (slurm " | + | |
< | < | ||
- | comp72: | + | comp72/06/01: |
- | comp06: | + | gpu72/06: |
- | comp01: | + | agpu72/06: |
- | gpu72: | + | himem72/06: |
- | gpu06: gpu nodes: 6 hour limit, 19 nodes | + | pubcondo06: |
- | himem72: | + | pcon06: |
- | himem06: | + | cloud72: |
- | pubcondo06: condo nodes all-user use, 6 hour limit, various constraints required, 25 nodes | + | condo: |
- | pcon06: | + | tres72/ |
- | cloud72: | + | razr72/ |
- | condo: | + | |
</ | </ | ||
- | + | ===Transition from Torque/PBS=== | |
- | ===Selecting the right Queue/Partition among multiple clusters=== | + | Basic slurm commands |
- | + | ||
- | Generally the nodes are reserved for the most efficient use, especially for expensive features such as GPU and extra memory. | + | |
- | Pinnacle compute nodes are reserved | + | |
- | Non-scalable programs should be run on Razor/Trestles (unless the 128GB shared memory size on Pinnacle is needed). | + | |
- | GPU nodes are reserved for programs that use the GPU. | + | |
- | Large memory nodes are reserved for programs that use more shared memory than is available on standard nodes (that is 128 to 768 GB). | + | |
- | Condo and pubcondo partitions require constraints | + | |
- | Possible constraints are | + | |
< | < | ||
- | 0gpu/ | + | sbatch |
- | avx2, | + | srun |
- | </ | + | squeue |
- | + | squeue | |
- | Jobs that don't meet these standards may be canceled without warning. | + | scancel |
- | Basic commands are, with transition from Torque/ | + | sinfo shownodes -l -n;qstat -q node status;list of queues |
- | + | ||
- | < | + | |
- | qsub | + | |
- | qsub -I srun | + | |
- | qstat squeue | + | |
- | qstat -u rfeynman | + | |
- | qdel | + | |
- | shownodes -l -n;qstat -q | + | |
</ | </ | ||
Line 65: | Line 44: | ||
< | < | ||
- | tres-l1:$ cat script.sh | + | pinnacle-l1:$ cat pbsscript.sh |
#PBS -N espresso | #PBS -N espresso | ||
#PBS -j oe | #PBS -j oe | ||
Line 81: | Line 60: | ||
- | tres-l1:$ pbs2slurm.sh | + | pinnacle-l1:$ pbs2slurm.sh |
- | tres-l1:$ cat script.slurm | + | pinnacle-l1:$ cat slurmscript.sh |
#!/bin/bash | #!/bin/bash | ||
#SBATCH --job-name=espresso | #SBATCH --job-name=espresso | ||
Line 120: | Line 99: | ||
Leading hash-bang /bin/sh or /bin/bash or /bin/tcsh is optional in torque, required in slurm, pbs2slurm.sh inserts it if not present | Leading hash-bang /bin/sh or /bin/bash or /bin/tcsh is optional in torque, required in slurm, pbs2slurm.sh inserts it if not present | ||
- | Use full nodes only on all-user Pinnacle (tasks-per-node=32 standard and 24 himem) except the cloud partition where single cores are available. | + | |
- | All non-cloud jobs should use either all the cores or more than 64 GB of memory, otherwise use Razor/ | + | Slurm does not autogenerate |
- | If using for the memory allocate all the cores anyway so that the node will not be split by the scheduler. | + | |
- | Valid condo (not pcon06) jobs may subdivide the nodes (tasks-per-node = integer divisions of 32/24). | + | |
- | Slurm does not autogenerate | + | |
< | < | ||
- | The generated machinefile differs from torque machinefile in that it has 1 entry per host instead of '' | + | The generated machinefile differs from torque machinefile in that it has 1 entry per host instead of '' |
- | Slurm does define a variable with the total number of cores '' | + | Slurm defines variables '' |
=== Interactive Jobs in SLURM === | === Interactive Jobs in SLURM === | ||
- | Multiple nodes or multiple tasks are not currently supported under srun, multiple cores up to the number in one node are. | ||
< | < | ||
Line 157: | Line 132: | ||
</ | </ | ||
- | ==Notes:== | + | === Software=== |
- | + | Modules are the same as on the Trestles cluster. We recommend the more recent versions of compiler | |
- | Condo/ | + | |
- | Similarly to razor/ | + | |
< | < | ||
- | Wang (20 nodes) | + | module load intel/20.0.1 mkl/20.0.1 impi/20.0.1 |
- | Alverson | + | module load gcc/9.3.1 |
- | Kaman | + | |
- | Bernhardt Barry ( 2 nodes) | + | |
- | + | ||
- | requesting a non-gpu condo node: | + | |
- | '' | + | |
- | '' | + | |
- | '' | + | |
- | requesting the gpu condo node: | + | |
- | '' | + | |
</ | </ | ||
- | === Software=== | + | ===Selecting the right Queue/ |
- | Modules are the same as on the Trestles cluster. We recommend the latest versions of compiler and math libraries so that they will recognize the AVX512 floating-point instructions. Examples: | + | |
+ | Generally the nodes are reserved for the most efficient use, especially for expensive features such as GPU and extra memory. | ||
+ | **Pinnacle compute nodes are** very busy (comp.. and himem.. partitions) are reserved for scalable **programs that can use all 32/24 cores** (except for the cloud partition, and condo usage by the owner). | ||
+ | Cores are allocated by the product of ntasks-per-node x cpus-per-task. | ||
+ | Exceptions: | ||
+ | (1) serial/ | ||
+ | (2) multiple jobs submitted together that use a whole node, such as 4 x 8 cores | ||
+ | (3) two jobs on one high-memory node (2 x 12 cores) that each use more than 192 GB (and less than 384 GB so that they can run on the himem node) | ||
+ | |||
+ | **Single core serial jobs should be run on the cloud.. partitions or tres.. or razr.. partitions** (unless requiring 64 to 192 GB, then run in the comp.. partitions with 32 cores allocated. | ||
+ | |||
+ | **GPU nodes are reserved for programs that use the GPU** (usually through the cuda libraries). | ||
+ | |||
+ | **Large memory nodes are reserved for programs that use more shared memory than the 192 GB** available on standard nodes. | ||
+ | |||
+ | **Condo jobs must have the id of the project PI/node owner as a constraint and unique node identifying information where the PI has more than one type.** | ||
+ | |||
+ | **Pubcondo non-gpu jobs must have** 0gpu as a constraint and the number of cores and memory as a constraint, with the memory reasonably related to the job. Options are 16c & 64gb (64 Intel nodes), 32c & 192gb (20 Intel nodes) , 32c & 256 gb (2 AMD nodes), 40c & 384 gb (10 Intel nodes), 48c & 256 gb (1 AMD node), 64c & 112 gb (2 Intel Phi nodes), 64c & 256 gb (5 AMD nodes) , 64c & 512gb (5 AMD nodes), 64c & 1024gb (1 AMD node), 64c & 2048 gb (1 AMD node). A slurm string would look like '' | ||
< | < | ||
- | module load intel/18.0.2 mkl/18.0.2 impi/18.0.2 | + | pinnacle-l1: |
- | module load gcc/7.3.1 | + | srun: job 706884 queued and waiting for resources |
+ | srun: job 706884 has been allocated resources | ||
+ | c3204: | ||
+ | |||
+ | pinnacle-l3: | ||
+ | srun: job 706892 queued and waiting for resources | ||
+ | srun: job 706892 has been allocated resources | ||
+ | c1522: | ||
</ | </ | ||
+ | |||
+ | **Pubcondo gpu jobs must have** the gpu type as a constraint and use that many gpus. Options are 4titanv & 24c (1 node), 1v100 & 40c (1 node), 2v100 & 32c (1 node), 1a100 & 64c (2 nodes), 4a100 & 64c (9 nodes). | ||
+ | |||
+ | ===Selecting cores per node=== | ||
+ | |||
+ | To maintain throughput and avoid wasting capacity with partly-filled nodes, there are standards for selecting part node/full node jobs on comp.., | ||
+ | |||
+ | Permitted exceptions: | ||
+ | multiple gpu.. jobs meant to share the gpu (set cores per job to set the number of jobs per node as for comp). | ||
+ | |||
+ | Jobs that don't meet these standards may be canceled without warning. |