===== Compression/Decompression =====
Using gzip, zip, pigz with 8 threads, and crabz with 8 threads on a 6.5 GB test file, one of the Blast NR tar files. crabz is very much faster for compression than the alternatives.
method compress uncompress
gzip 335 s 74 s
zip 315 s 70 s
pgzip 41 s 43 s
crabz 21 s 50 s
gzip: gzip nr.22.tar, gunzip nr.22.tar
zip: zip -r nr.22.tar.zip nr.22.tar, unzip nr.22.tar.zip
pgzip: /share/apps/pigz/pigz-2.7/pigz -p 8 nr.22.tar, /share/apps/pigz/pigz-2.7/pigz -p 8 -d nr.22.tar.gz
crabz: crabz -p 8 -I nr.22.tar, crabz -d -p 8 -I nr.22.tar.gz
crabz startup: module load python/anaconda-3.8; source /share/apps/bin/conda-3.8.sh; conda activate crabz