=====Slurm Queues Pinnacle/Karpinski===== See [[ equipment | Selecting Resources ]] for help on choosing the best node/queue for your work. Updates: tres288 queue added with 288 hour/12 day maximum tres72 time limit changed to 288 hours, same as tres288, retained for existing scripts csce-k2-72 queue added for new csce Pinnacle-2 nodes Pinnacle queues or ''slurm'' "partitions" are: pinnacle partition,description,time limit,cores per node,number of nodes,other comp01,192 GB nodes,1 hr, 32, 48,full node usage required comp06,192 GB nodes,6 hr, 32, 44,full node usage required comp72,192 GB nodes,72 hr, 32, 40,full node usage required gpu06,gpu nodes,6 hr,32, 19,gpu usage required/full node usage required gpu72,gpu nodes,72 hr,32, 19,gpu usage required/full node usage required himem06,768 GB nodes, 6 hr,24, 6,>192 GB memory usage required/full node usage required himem72,768 GB nodes,72 hr,24, 6,>192 GB memory usage required/full node usage required cloud72,virtual machines/containers/single processor jobs, 72 hr, 32, 3,for non-intensive computing up to 4 cores tres72, 64 GB nodes, 72hr, 32, 23, Trestles nodes with Pinnacle operating system tres288, 64 GB nodes, 288hr, 32, 23, Trestles nodes with Pinnacle operating system karpinski partition,description,time limit,cores per node,number of nodes csce72,32 GB nodes, 72 hr,8, 18 csce-k2-72, 256 GB nodes, 72 hr, 64, 6 cscloud72,virtual machines/containers/single processor jobs, 72 hr,8, 18 Condo queues are: pinnacle partition,description,time limit,number of nodes,other condo,condo nodes, none,25, authorization and appropriate properties required pcon06,public use of condo nodes,6 hr, 25, appropriate properties required Condo nodes require specification of a sufficient set of slurm properties. Property choices available are: **condo/pcon06 jobs running on the wrong nodes through lack of specified properties will be canceled without notice**\\ **non-gpu jobs running on gpu nodes may be canceled without notice**\\ gpu or not: ''0gpu''/''1v100''/''2v100''/''1a100''/''4a100''\\ processor: ''i6130''/''a7351''/''i6128''\\ equivalently: ''192gb''/''256gb''/''768gb''\\ equivalently: ''32c''/''32c''/''24c''\\ local drive: ''nvme''/no specification\\ research group: ''fwang'' equivalent to ''0gpu''/''i6130|i6230''/''768gb''/''32c|40c''/nvme\\ research group: ''tkaman'' equivalent to ''2v100''/''i6130''/''192gb''/''32c''\\ research group: ''aja'' equivalent to ''0gpu''/''i6128''/''192gb|768gb''/''24c'' examples:\\ ''#SBATCH --constraint=2v100''\\ ''#SBATCH --constraint=fwang''\\ ''#SBATCH --constraint=768gb&0gpu''\\ ''#SBATCH --constraint=256gb'' A script is available to show idle nodes like this (in this case 2 nodes idle in the1-hour comp queue, none in 6-hour or 72-hour comp queue, but nodes available in gpu, himem, csce, and csce cloud. Sufficient idle nodes in your queue of interest do not guarantee that your job will start immediately, but that is usually the case. $ idle_pinnacle_nodes.sh n01=2 n06=0 n72=0 g06=1 g72=1 h06=2 h72=2 c72=16 l72=16 condo aja=2 wang=0 mich=2 kama=0 $ === Public Condo Queue - pcon06 === The condo nodes, which are reserved for priority access by the condo node owners, are also available for public use via the **''pcon06''** queue. There is to 6 hour walltime limit for **''pcon06''**, but it may be extended upon request if there are no condo owner jobs waiting in the queue. The **''pcon06''** contains a collection of multiple types of nodes purchased by different departments at various times. So the hardware configuration for those nodes varies. Each node in the queue has a set of features assigned to it which describe its hardware. To select the appropriate node, slurm uses a **constraints** (''-C'') parameter in the **''sbatch''** and **''srun''** commands. **''pcon06-info.sh''** script is available to list the available idle nodes in the **''pcon06''** queue along with a list of constraints for each node. pinnacle-l5:pwolinsk:~$ pcon06-info.sh Idle pcon06 nodes NodeName Constraint list ============================ c1302: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1305: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1306: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1307: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1308: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1309: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1310: fwang,0gpu,nvme,384gb,i6230,avx512,40c,intel c1311: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1312: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1313: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1314: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1315: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1316: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1317: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1318: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1319: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1320: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1321: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1322: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1323: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1324: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1325: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1326: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1328: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1329: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1330: fwang,0gpu,nvme,192gb,i6130,avx512,32c,intel c1432: aja,0gpu,256gb,a7543,avx2,64c,amd c1618: jzhao77,0gpu,256gb,a7402,avx2,48c,amd c1716: yongwang,1v100,192gb,i6230,avx512,40c,intel c1719: mlbernha,0gpu,256gb,a7351,avx2,32c,amd c1720: mlbernha,0gpu,256gb,a7351,avx2,32c,amd c1913: laurent,0gpu,256gb,a7543,avx2,64c,amd c1915: laurent,0gpu,256gb,a7543,avx2,64c,amd c1916: laurent,0gpu,256gb,a7543,avx2,64c,amd c1917: laurent,0gpu,256gb,a7543,avx2,64c,amd c1918: laurent,0gpu,256gb,a7543,avx2,64c,amd c1919: laurent,0gpu,256gb,a7543,avx2,64c,amd c1920: laurent,0gpu,256gb,a7543,avx2,64c,amd c2001: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2002: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2003: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2004: aimrc,4a100,1024gb,a7543,avx2,64c,amd c2010: zhang,2a100,512gb,a7543,avx2,64c,amd c2011: harris,0gpu,1024gb,a7543,avx2,64c,amd c2101: csce,4a100,1024gb,a7543,avx2,64c,amd c2102: csce,4a100,1024gb,a7543,avx2,64c,amd c2103: csce,4a100,1024gb,a7543,avx2,64c,amd c2104: csce,4a100,1024gb,a7543,avx2,64c,amd c2105: harris,4a100,1024gb,a7543,avx2,64c,amd c2112: kmbefus,0gpu,1024gb,a7543,avx2,64c,amd c2113: fwang,0gpu,512gb,a7543,avx2,64c,amd c2114: fwang,0gpu,512gb,a7543,avx2,64c,amd c2115: fwang,0gpu,512gb,a7543,avx2,64c,amd c2116: fwang,0gpu,512gb,a7543,avx2,64c,amd c2118: jm217,1a100,1024gb,a7543,avx2,64c,amd c2402: kwalters,1a40,1024gb,a7543,avx2,64c,amd c2403: kwalters,1a40,1024gb,a7543,avx2,64c,amd c2404: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2405: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2406: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2407: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2408: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2409: kmbefus,0gpu,1024gb,a7543,avx2,64c,amd c2416: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2417: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2418: kwalters,0gpu,256gb,a7543,avx2,64c,amd c2421: laurent,0gpu,256gb,a7543,avx2,64c,amd c2422: laurent,0gpu,256gb,a7543,avx2,64c,amd c2423: laurent,0gpu,256gb,a7543,avx2,64c,amd c3101: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3103: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3104: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3107: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3108: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3109: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3110: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3111: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3114: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3115: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3116: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3118: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3119: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3120: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3121: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3122: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3123: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3124: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3125: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3126: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3127: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3128: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3129: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3130: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3131: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3132: pmillett,0gpu,64gb,i2650v2,avx,16c,intel c3133: pmillett,4k80,128gb,i2650v2,avx,16c,intel c3201: nair,0gpu,64gb,i2650v2,avx,16c,intel c3202: nair,0gpu,64gb,i2650v2,avx,16c,intel c3203: nair,0gpu,64gb,i2650v2,avx,16c,intel c3204: nair,0gpu,64gb,i2650v2,avx,16c,intel c3205: nair,0gpu,64gb,i2650v2,avx,16c,intel c3206: nair,0gpu,64gb,i2650v2,avx,16c,intel c3207: nair,0gpu,64gb,i2650v2,avx,16c,intel c3208: nair,0gpu,64gb,i2650v2,avx,16c,intel c3209: nair,0gpu,64gb,i2650v2,avx,16c,intel c3210: nair,0gpu,64gb,i2650v2,avx,16c,intel c3211: nair,0gpu,64gb,i2650v2,avx,16c,intel c3212: nair,0gpu,64gb,i2650v2,avx,16c,intel c3213: nair,0gpu,64gb,i2650v2,avx,16c,intel c3214: nair,0gpu,64gb,i2650v2,avx,16c,intel c3216: nair,0gpu,64gb,i2650v2,avx,16c,intel c3217: nair,0gpu,64gb,i2650v2,avx,16c,intel c3219: nair,0gpu,64gb,i2650v2,avx,16c,intel c3220: nair,0gpu,64gb,i2650v2,avx,16c,intel c3221: nair,0gpu,64gb,i2650v2,avx,16c,intel c3222: nair,0gpu,64gb,i2650v2,avx,16c,intel c3224: nair,0gpu,64gb,i2650v2,avx,16c,intel c3226: nair,0gpu,64gb,i2650v2,avx,16c,intel c3227: nair,0gpu,64gb,i2650v2,avx,16c,intel example submit commands: srun -p pcon06 -t 6:00:00 -n 16 -q comp -C 'nair&0gpu&64gb&i2650v2&avx&16c&intel' --pty /bin/bash sbatch -p pcon06 -t 6:00:00 -n 16 -q comp -C 'nair&0gpu&64gb&i2650v2&avx&16c&intel' .slrum pinnacle-l5:pwolinsk:~$ srun -p pcon06 -t 6:00:00 -N 2 -n 16 -q comp -C 'nair&0gpu&64gb&i2650v2&avx&16c&intel' --pty /bin/bash srun: job 338300 queued and waiting for resources srun: job 338300 has been allocated resources c3201:pwolinsk:~$