Table of Contents

Condo Queues

The Condo program allows individual users to purchase computing hardware with their own funds and use AHPCC staff to provide installation, power, cooling, networking and maintenance services. The condo hardware owner has priority access to the condo nodes and has no job walltime restriction. However, the condo nodes are also available for public use for up to 6 hour walltime jobs. The 6 hour limit may be extended upon request if there are no condo owner jobs waiting in the queue.

All condo nodes are accessible via two queues: qcondo_public and qcondo. The qcondo-public queue is unrestricted and available for all users. The qcondo queue is restricted and available for use by condo node owners and their research groups only.

Trestles

To get a listing of all condo nodes on Trestles and their properties:

tres-l1:pwolinsk:$ for i in `node-queue-map |grep qcondo_public |cut -d" " -f3-`; do echo -n "$i "; pbsnodes $i |grep properties; done
tres0733      properties = mib,douglas,m768gb
tres0734      properties = mib,douglas,v4,m256gb
tres0735      properties = mib,douglas,v4,m256gb
tres0736      properties = mib,douglas,v4,m256gb
tres0833      properties = mib,abi,m3072gb
tres0834      properties = mib,laurent,v4,m64gb
tres0835      properties = mib,laurent,v4,m64gb
tres0836      properties = mib,laurent,v4,m64gb
tres-l1:pwolinsk:$ 

The properties can be specified following the ppn= directive, i.e. qsub -l nodes=1:ppn=1:<property> . To submit a job to qcondo_public queue on trestles to use 1 node with property m64gb (64 GB or RAM):

tres-l1:pwolinsk:$ qsub -q qcondo_public -l walltime=6:00:00 -l nodes=1:ppn=32:m64gb hostname.pbs
283336.torque
tres-l1:pwolinsk:$ 

Razor

To get a listing of all condo nodes on Razor and their properties:

razor-l1:pwolinsk:$ for i in `node-queue-map |grep qcondo |cut -d" " -f3-`; do echo -n "$i "; pbsnodes $i |grep properties; done
compute0805      properties = mib,ib,gpu,k40gpu,m64gb
compute1153      properties = qib,ib,laurent,m24gb
compute1154      properties = qib,ib,laurent,m24gb
compute1155      properties = qib,ib,laurent,m24gb
...
compute3131      properties = mib,ib,pmillett,m64gb
compute3132      properties = mib,ib,pmillett,m64gb
compute3133      properties = mib,ib,v4,gpu,k80gpu,pmillettgpu,m128gb
compute3134      properties = mib,ib,v4,gpu,k80gpu,pmillettgpu,m128gb
compute3135      properties = mib,ib,v4,gpu,k80gpu,pmillettgpu,m128gb
...
compute3231      properties = mib,ib,nair,m64gb
compute3232      properties = mib,ib,nair,m64gb
razor-l1:pwolinsk:$ 

qcondo_public - Unrestricted Access

The qcondo_public queue can be used by all users. To get a listing of the available nodes in that queue on Trestles:

tres-l1:pwolinsk:$ node-queue-map |grep qcondo_public
qcondo_public 8 tres0733 tres0734 tres0735 tres0736 tres0833 tres0834 tres0835 tres0836 

In a regular (non-condo) queue all nodes are identical; have same cpus, memory, etc. - have the same properites. The qcondo_public queue is different. Individual nodes where purchased by different users and have different properties. To get a listing of properties for an individual node:

tres-l1:pwolinsk:$ pbsnodes tres0733 |grep properties
     properties = mib,douglas,m768gb

Below is an example of submitting a job to qcondo_public queue on Razor to use 2 node with property k80gpu (node with Nvidia K80 GPU):

razor-l1:pwolinsk:$ qsub -q qcondo_public -l nodes=2:ppn=24:k80gpu hostname.pbs 
1774072.sched

qcondo - Restricted Access

The qcondo queue should only be used by condo node owners and their research groups. To submit jobs to this queue the owner property must be specified during job submission. Below is an example of submitting a job to one of the nodes in the qcondo queue which belongs to the “douglas” condo group:

tres-l1:pwolinsk:$ qsub -q qcondo -l walltime=200:00:00 -l nodes=1:ppn=32:douglas hostname.pbs
283336.torque
tres-l1:pwolinsk:$ 

NOTE: the owner property must be present when submitting to the qcondo queue.

Property Code Summary

Property codeMeaning
ibInfiniband equipped node
mibMellanox Ininiband card
qibQLogic Infiniband card
abicondo owner code: Arkansas Bioinformatics Institute
douglascondo owner code: Douglas group
itzanetacondo owner code: Tzanetakis group
laurentcondo owner code: Bellaiche group
naircondo owner code: Nair group
pmilletcondo owner code: Millet group
xqiancondo owner code: Qian group:w
v4Xeon Broadwell cpu
m64gbnode memory size 64GB
m128gbnode memory size 128GB
m256gbnode memory size 256GB
m512gbnode memory size 512GB
m768GBnode memory size 768GB
3072GBnode memory size 3TB
gpunode equipped with a GPU card
k40gpunode equipped with Nvidia K40 GPU(s)
k80gpunode equipped with Nvidia K80 GPU(s)