See Selecting Resources for help on choosing the best node/queue for your work.
Updates:
tres288 queue added with 288 hour/12 day maximum tres72 time limit changed to 288 hours, same as tres288, retained for existing scripts csce-k2-72 queue added for new csce Pinnacle-2 nodes
Pinnacle queues or slurm
“partitions” are:
pinnacle partition | description | time limit | cores per node | number of nodes | other |
---|---|---|---|---|---|
comp01 | 192 GB nodes | 1 hr | 32 | 48 | full node usage required |
comp06 | 192 GB nodes | 6 hr | 32 | 44 | full node usage required |
comp72 | 192 GB nodes | 72 hr | 32 | 40 | full node usage required |
gpu06 | gpu nodes | 6 hr | 32 | 19 | gpu usage required/full node usage required |
gpu72 | gpu nodes | 72 hr | 32 | 19 | gpu usage required/full node usage required |
himem06 | 768 GB nodes | 6 hr | 24 | 6 | >192 GB memory usage required/full node usage required |
himem72 | 768 GB nodes | 72 hr | 24 | 6 | >192 GB memory usage required/full node usage required |
cloud72 | virtual machines/containers/single processor jobs | 72 hr | 32 | 3 | for non-intensive computing up to 4 cores |
tres72 | 64 GB nodes | 72hr | 32 | 23 | Trestles nodes with Pinnacle operating system |
tres288 | 64 GB nodes | 288hr | 32 | 23 | Trestles nodes with Pinnacle operating system |
karpinski partition | description | time limit | cores per node | number of nodes |
---|---|---|---|---|
csce72 | 32 GB nodes | 72 hr | 8 | 18 |
csce-k2-72 | 256 GB nodes | 72 hr | 64 | 6 |
cscloud72 | virtual machines/containers/single processor jobs | 72 hr | 8 | 18 |
Condo queues are:
pinnacle partition | description | time limit | number of nodes | other |
---|---|---|---|---|
condo | condo nodes | none | 25 | authorization and appropriate properties required |
pcon06 | public use of condo nodes | 6 hr | 25 | appropriate properties required |
Condo nodes require specification of a sufficient set of slurm properties. Property choices available are:
condo/pcon06 jobs running on the wrong nodes through lack of specified properties will be canceled without notice
non-gpu jobs running on gpu nodes may be canceled without notice
gpu or not: 0gpu
/1v100
/2v100
/1a100
/4a100
processor: i6130
/a7351
/i6128
equivalently: 192gb
/256gb
/768gb
equivalently: 32c
/32c
/24c
local drive: nvme
/no specification
research group: fwang
equivalent to 0gpu
/i6130|i6230
/768gb
/32c|40c
/nvme
research group: tkaman
equivalent to 2v100
/i6130
/192gb
/32c
research group: aja
equivalent to 0gpu
/i6128
/192gb|768gb
/24c
examples:
#SBATCH –constraint=2v100
#SBATCH –constraint=fwang
#SBATCH –constraint=768gb&0gpu
#SBATCH –constraint=256gb
A script is available to show idle nodes like this (in this case 2 nodes idle in the1-hour comp queue, none in 6-hour or 72-hour comp queue, but nodes available in gpu, himem, csce, and csce cloud. Sufficient idle nodes in your queue of interest do not guarantee that your job will start immediately, but that is usually the case.
$ idle_pinnacle_nodes.sh n01=2 n06=0 n72=0 g06=1 g72=1 h06=2 h72=2 c72=16 l72=16 condo aja=2 wang=0 mich=2 kama=0 $
The condo nodes, which are reserved for priority access by the condo node owners, are also available for public use via the pcon06
queue. There is to 6 hour walltime limit for pcon06
, but it may be extended upon request if there are no condo owner jobs waiting in the queue. The pcon06
contains a collection of multiple types of nodes purchased by different departments at various times. So the hardware configuration for those nodes varies. Each node in the queue has a set of features assigned to it which describe its hardware. To select the appropriate node, slurm uses a constraints (-C
) parameter in the sbatch
and srun
commands.
pcon06-info.sh
script is available to list the available idle nodes in the pcon06
queue along with a list of constraints for each node.
pinnacle-l5:pwolinsk:~$ pcon06-info.sh Idle pcon06 nodes NodeName Constraint list ============================ c1302: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1305: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1306: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1307: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1308: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1309: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1310: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1311: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1312: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1313: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1314: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1315: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1316: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1317: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1318: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1319: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1320: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1321: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1322: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1323: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1324: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1325: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1326: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1328: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1329: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1330: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1432: aja,0gpu,256gb,a7543,avx2,64c,amd c1618: jzhao77,0gpu,256gb,a7402,avx2,48c,amd c1716: yongwang,1v100,192gb,i6230,avx512,40c,intel c1719: mlbernha,0gpu,256gb,a7351,avx2,32c,amd c1720: mlbernha,0gpu,256gb,a7351,avx2,32c,amd c1913: laurent,0gpu,256gb,a7543,avx2,64c,amd c1915: laurent,0gpu,256gb,a7543,avx2,64c,amd c1916: laurent,0gpu,256gb,a7543,avx2,64c,amd c1917: laurent,0gpu,256gb,a7543,avx2,64c,amd c1918: laurent,0gpu,256gb,a7543,avx2,64c,amd c1919: laurent,0gpu,256gb,a7543,avx2,64c,amd c1920: laurent,0gpu,256gb,a7543,avx2,64c,amd c2001: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2002: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2003: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2004: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2010: zhang,2a100,512gb,a7543,avx2,64c,amd c2011: harris,0gpu,1024gb,a7543,avx2,64c,amd c2101: csce,4a100,1024gb,a7543,avx2,64c,amd c2102: csce,4a100,1024gb,a7543,avx2,64c,amd c2103: csce,4a100,1024gb,a7543,avx2,64c,amd c2104: csce,4a100,1024gb,a7543,avx2,64c,amd c2105: harris,4a100,1024gb,a7543,avx2,64c,amd c2112: kmbefus,0gpu,1024gb,a7543,avx2,64c,amd c2113: fwang,0gpu,512gb,a7543,avx2,64c,amd c2114: fwang,0gpu,512gb,a7543,avx2,64c,amd c2115: fwang,0gpu,512gb,a7543,avx2,64c,amd c2116: fwang,0gpu,512gb,a7543,avx2,64c,amd c2118: jm217,1a100,1024gb,a7543,avx2,64c,amd c2402: kwalters,1a40,1024gb,a7543,avx2,64c,amd c2403: kwalters,1a40,1024gb,a7543,avx2,64c,amd c2404: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2405: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2406: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2407: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2408: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2409: kmbefus,0gpu,1024gb,a7543,avx2,64c,amd c2416: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2417: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2418: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2421: laurent,0gpu,256gb,a7543,avx2,64c,amd c2422: laurent,0gpu,256gb,a7543,avx2,64c,amd c2423: laurent,0gpu,256gb,a7543,avx2,64c,amd c3101: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3103: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3104: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3107: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3108: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3109: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3110: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3111: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3114: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3115: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3116: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3118: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3119: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3120: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3121: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3122: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3123: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3124: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3125: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3126: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3127: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3128: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3129: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3130: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3131: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3132: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3133: pmillett,4k80,128gb,i2650v2,avx,16c,intel c3201: nair,0gpu,64gb,i2650v2,avx,16c,intel c3202: nair,0gpu,64gb,i2650v2,avx,16c,intel c3203: nair,0gpu,64gb,i2650v2,avx,16c,intel c3204: nair,0gpu,64gb,i2650v2,avx,16c,intel c3205: nair,0gpu,64gb,i2650v2,avx,16c,intel c3206: nair,0gpu,64gb,i2650v2,avx,16c,intel c3207: nair,0gpu,64gb,i2650v2,avx,16c,intel c3208: nair,0gpu,64gb,i2650v2,avx,16c,intel c3209: nair,0gpu,64gb,i2650v2,avx,16c,intel c3210: nair,0gpu,64gb,i2650v2,avx,16c,intel c3211: nair,0gpu,64gb,i2650v2,avx,16c,intel c3212: nair,0gpu,64gb,i2650v2,avx,16c,intel c3213: nair,0gpu,64gb,i2650v2,avx,16c,intel c3214: nair,0gpu,64gb,i2650v2,avx,16c,intel c3216: nair,0gpu,64gb,i2650v2,avx,16c,intel c3217: nair,0gpu,64gb,i2650v2,avx,16c,intel c3219: nair,0gpu,64gb,i2650v2,avx,16c,intel c3220: nair,0gpu,64gb,i2650v2,avx,16c,intel c3221: nair,0gpu,64gb,i2650v2,avx,16c,intel c3222: nair,0gpu,64gb,i2650v2,avx,16c,intel c3224: nair,0gpu,64gb,i2650v2,avx,16c,intel c3226: nair,0gpu,64gb,i2650v2,avx,16c,intel c3227: nair,0gpu,64gb,i2650v2,avx,16c,intel example submit commands: srun -p pcon06 -t 6:00:00 -n 16 -q comp -C 'nair&0gpu&64gb&i2650v2&avx&16c&intel' --pty /bin/bash sbatch -p pcon06 -t 6:00:00 -n 16 -q comp -C 'nair&0gpu&64gb&i2650v2&avx&16c&intel' <slurm_script>.slrum pinnacle-l5:pwolinsk:~$ srun -p pcon06 -t 6:00:00 -N 2 -n 16 -q comp -C 'nair&0gpu&64gb&i2650v2&avx&16c&intel' --pty /bin/bash srun: job 338300 queued and waiting for resources srun: job 338300 has been allocated resources c3201:pwolinsk:~$