| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| storage [2020/02/04 20:06] – root | storage [2025/10/15 19:51] (current) – external edit 127.0.0.1 |
|---|
| |
| ==Lab Storage== | ==Lab Storage== |
| Some labs have purchased auxiliary storage and if you are part of that corresponding lab group you can have a directory on it. These have names such as ''storageb''. If your lab has some storage, they are suitable for moving over-quota files or for backing up your ''/scrfs/storage/'' area. | Some labs have purchased auxiliary storage and if you are part of that corresponding lab group you can have a directory on it. These have names such as ''/storageb/''. If your lab has some storage, they are suitable for moving over-quota files or for backing up your ''/scrfs/storage/'' area. |
| |
| ==scratch and local_scratch== | ==scratch and local_scratch== |
| There is a dedicated and small high-speed temporary storage called ''/scratch''. This is intended for large inputs (and especially outputs) directly to computational jobs. There is also local disk storage on each compute node called ''/local_scratch''. The job queueing system creates for each job temporary job directories ''/scratch/$SLURM_JOB_ID'' and ''/local_scratch/$SLURM_JOB_ID'' on the first compute node of the job. On torque systems ''$PBS_JOBID'' is used instead. | There is a dedicated and small high-speed temporary storage called ''/scratch/''. This is intended for large inputs (and especially outputs) directly to computational jobs. There is also local disk storage on each compute node called ''/local_scratch/''. The job queueing system creates for each job temporary job directories ''/scratch/$SLURM_JOB_ID/'' and ''/local_scratch/$SLURM_JOB_ID/'' on the first compute node of the job. On torque systems ''$PBS_JOBID'' is used instead. |
| |
| If your job creates more than 500 MB of output, please route output to the job scratch or local_scratch directory. There are no quotas on ''/scratch'' or ''/local_scratch'', but ''/scratch'' has a total size of 19 TB and ''/local_scratch'' varies by node but may be as small as 90 GB. The purpose for this rerouting is performance. The main storage ''/scrfs'' is composed of fairly large and slow 8 TB SATA drives that do not well handle hundreds of streams of data, particularly those with small data blocks. ''/scrfs'' can handle a fairly large throughput of large-blocked data, but that is rare in application programs. The NVMe drives of ''/scratch'' and the mostly SSD drives of ''/local_scratch'' are better for the typically small-blocked data put out by programs. At the end of your job, copy the files you want to keep from ''/scratch'' or ''/local_scratch'' back to main storage ''/scrfs''. | If your job creates more than 500 MB of output, please route output to the job scratch or local_scratch directory. There are no quotas on ''/scratch/'' or ''/local_scratch/'', but ''/scratch/'' has a total size of 19 TB and ''/local_scratch/'' varies by node but may be as small as 90 GB. The purpose for this rerouting is performance. The main storage ''/scrfs/'' is composed of fairly large and slow 8 TB SATA drives that do not well handle hundreds of concurrent streams of data, particularly those with small data blocks. ''/scrfs/'' can handle a fairly large throughput of efficiently large-blocked data, but that is rare in application programs. The NVMe drives of ''/scratch/'' and the mostly SSD drives of ''/local_scratch/'' are better for the typically inefficient and small data blocksput out by programs. At the end of your job, copy the files you want to keep from ''/scratch/'' or ''/local_scratch'/' back to main storage ''/scrfs/''. |
| |
| There are no user directories such as ''/scratch/rfeynman/'' since we found that such directories soon filled the small ''/scratch'' partition. Each job directory is normally retained until a week after the job ends unless space becomes critical. | There are no user directories such as ''/scratch/rfeynman/'' since we found that such directories soon filled the small ''/scratch/'' partition. Each job directory is normally retained until a week after the job ends unless space becomes critical. |
| |
| See [[ torque_slurm_scripts ]] for some hints on moving data into and out of ''/scratch'' areas during jobs. | See [[ torque_slurm_scripts ]] for some hints on moving data into and out of ''/scratch/'' areas during jobs. |
| |
