The Condo program allows individual users to purchase computing hardware with their own funds and use AHPCC staff to provide installation, power, cooling, networking and maintenance services. The condo hardware owner has priority access to the condo nodes and has no job walltime restriction. However, the condo nodes are also available for public use for up to 6 hour walltime jobs. The 6 hour limit may be extended upon request if there are no condo owner jobs waiting in the queue.

All condo nodes are accessible via two queues: qcondo_public and qcondo. The qcondo-public queue is unrestricted and available for all users. The qcondo queue is restricted and available for use by condo node owners and their research groups only.

To get a listing of all condo nodes on Trestles and their properties:

tres-l1:pwolinsk:$ for i in `node-queue-map |grep qcondo_public |cut -d" " -f3-`; do echo -n "$i "; pbsnodes $i |grep properties; done
tres0733      properties = mib,douglas,m768gb
tres0734      properties = mib,douglas,v4,m256gb
tres0735      properties = mib,douglas,v4,m256gb
tres0736      properties = mib,douglas,v4,m256gb
tres0833      properties = mib,abi,m3072gb
tres0834      properties = mib,laurent,v4,m64gb
tres0835      properties = mib,laurent,v4,m64gb
tres0836      properties = mib,laurent,v4,m64gb
tres-l1:pwolinsk:$ 

The properties can be specified following the ppn= directive, i.e. qsub -l nodes=1:ppn=1:<property> . To submit a job to qcondo_public queue on trestles to use 1 node with property m64gb (64 GB or RAM):

tres-l1:pwolinsk:$ qsub -q qcondo_public -l walltime=6:00:00 -l nodes=1:ppn=32:m64gb hostname.pbs
283336.torque
tres-l1:pwolinsk:$ 

To get a listing of all condo nodes on Razor and their properties:

razor-l1:pwolinsk:$ for i in `node-queue-map |grep qcondo |cut -d" " -f3-`; do echo -n "$i "; pbsnodes $i |grep properties; done
compute0805      properties = mib,ib,gpu,k40gpu,m64gb
compute1153      properties = qib,ib,laurent,m24gb
compute1154      properties = qib,ib,laurent,m24gb
compute1155      properties = qib,ib,laurent,m24gb
...
compute3131      properties = mib,ib,pmillett,m64gb
compute3132      properties = mib,ib,pmillett,m64gb
compute3133      properties = mib,ib,v4,gpu,k80gpu,pmillettgpu,m128gb
compute3134      properties = mib,ib,v4,gpu,k80gpu,pmillettgpu,m128gb
compute3135      properties = mib,ib,v4,gpu,k80gpu,pmillettgpu,m128gb
...
compute3231      properties = mib,ib,nair,m64gb
compute3232      properties = mib,ib,nair,m64gb
razor-l1:pwolinsk:$ 

The qcondo_public queue can be used by all users. To get a listing of the available nodes in that queue on Trestles:

tres-l1:pwolinsk:$ node-queue-map |grep qcondo_public
qcondo_public 8 tres0733 tres0734 tres0735 tres0736 tres0833 tres0834 tres0835 tres0836 

In a regular (non-condo) queue all nodes are identical; have same cpus, memory, etc. - have the same properites. The qcondo_public queue is different. Individual nodes where purchased by different users and have different properties. To get a listing of properties for an individual node:

tres-l1:pwolinsk:$ pbsnodes tres0733 |grep properties
     properties = mib,douglas,m768gb

Below is an example of submitting a job to qcondo_public queue on Razor to use 2 node with property k80gpu (node with Nvidia K80 GPU):

razor-l1:pwolinsk:$ qsub -q qcondo_public -l nodes=2:ppn=24:k80gpu hostname.pbs 
1774072.sched

The qcondo queue should only be used by condo node owners and their research groups. To submit jobs to this queue the owner property must be specified during job submission. Below is an example of submitting a job to one of the nodes in the qcondo queue which belongs to the “douglas” condo group:

tres-l1:pwolinsk:$ qsub -q qcondo -l walltime=200:00:00 -l nodes=1:ppn=32:douglas hostname.pbs
283336.torque
tres-l1:pwolinsk:$ 

NOTE: the owner property must be present when submitting to the qcondo queue.