There data transfer protocols are supported to move data to and from the main storage on the AHPCC clusters:
scp
(secure copy)sftp
(secure ftp)rsync
In addition the wget
and curl
commands are available to download data to your account using a public URL.
To upload a data file from the current directory on your local desktop machine to your /storage directory on pinnacle:
pawel@localdesktop$# scp localfile.dat pwolinsk@hpc-portal2.hpc.uark.edu:/storage/pwolinsk/
To download a data file from your /storage directory on pinnacle to the current directory on your local desktop machine:
pawel@localdesktop$# scp pwolinsk@hpc-portal2.hpc.uark.edu:/storage/pwolinsk/remotefile.dat .
Windows OS does not include Secure Copy or Secure FTP tools. There are multiple file transfer clients available for download. The most popular command line client is pscp.exe available here:
https://the.earth.li/~sgtatham/putty/latest/x86/pscp.exe
To transfer files using pscp.exe download and save pscp.exe to your Windows machine. Then open a terminal (i.e. Command Prompt, under “Start→All Programs→Accessories→Command Prompt”) and specify the full path to the location of the downloaded pscp.exe file followed by two arguments <source> and <destination>. Either one or both could specify a file a on remote machine (user@host:pathtofile) or a local file (pathtofile). For example:
C:\Users\Pawel> c:\Users\Pawel\Downloads\pscp.exe filetoupload.txt pwolinsk@hpc-portal2.hpc.uark.edu:
The code above uses secure copy protocol to upload a file “filetoupload.txt” to the home directory of user pwolinsk on hpc-portal2.hpc.uark.edu.
Another popular windows transfer client (GUI) is WinSCP:
https://winscp.net/eng/download.php
NOTE: Pinnacle, Trestles and Razor clusters all share the same main AHPCC storage. So transferring files to any one of the 3 clusters will make those files available on all clusters.
GLOBUS https://globus.org/ is a service for secure, reliable research data management. It allows users to move, share, & discover data via a single interface using a web browser. GLOBUS is designed to move large data sets.
scp, sftp, rsync have the advantage of being very simple and do not require any initial setup. However large data sets often fail to transfer correctly using these protocols. GLOBUS does require some initial setup but is much more reliable. It has many features, such as splitting the transfer into multiple simultaneous streams, encrypting the data in flight, automatically retransmitting data on network failure/timeouts, verifying data integrity after transfer. In addition our installation of GLOBUS is connected to the 100Gb/s network (while our cluster login nodes used for scp/sftp/rsync are on the 10Gb/s network).
GLOBUS service moves data between GLOBUS Endpoints. Each Endpoint is a server process running on a machine which can send and receive data. One such endpoint, named UARK-Pinnacle-2024, is set up on the Pinnacle cluster. It is a public endpoint (visible to all GLOBUS users), and accessible by anyone with an account on Pinnacle. To transfer data between your account on Pinnacle and your personal workstation/laptop you will need to set up a private GLOBUS endpoint, which is only visible and accessible by you. This requires the installation of GLOBUS personal connect server on your workstation/laptop.
To install GLOBUS personal connect server:
To transfer data between the private GLOBUS endpoint (which you just created) on your workstation/laptop and the public endpoint on Pinnacle we have to find and connect to both endpoints using the “File Manager” in the GLOBUS portal.
You are already connected to the public UARK-Pinnacle Endpoint and logged into your account on Pinnacle. To transfer files between your local workstation/laptop and UARK-Pinnacle-2024 endpoint, you will also need to find and open your private endopint. If you already see both 'UARK-Pinnacle-2024' and your private endpoint in the file manager in the GLOBUS portal you can drag and drop files between the windows in the file manager, which will start transfer of data. If not, you will have to find and connect to your private endpoint on your workstation/laptop:
With both endpoints connected you can now drag and drop files between the endpoints.