User Tools

Site Tools


moving_data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
moving_data [2016/03/09 21:07]
root
moving_data [2017/02/21 19:54] (current)
root
Line 6: Line 6:
    * **''​sftp''​** (secure ftp)  ​    * **''​sftp''​** (secure ftp)  ​
    * **''​rsync''​**    * **''​rsync''​**
 +
 +=== Linux and MacOS ===
  
 To upload a data file from the current directory on your local desktop machine to your /storage directory on **razor**: To upload a data file from the current directory on your local desktop machine to your /storage directory on **razor**:
Line 16: Line 18:
 </​code>​ </​code>​
 You will also have a staging directory on **tgv**/​**dm** called **/​local_storage/​$USER/​**. ​ A new Globus Online instance on **tgv**/​**dm** is in preparation,​ and login shells are available for special situations such as batch **''​wget''​** from an http server. ​ Please contact hpc-support@listserv.uark.edu if you need that. You will also have a staging directory on **tgv**/​**dm** called **/​local_storage/​$USER/​**. ​ A new Globus Online instance on **tgv**/​**dm** is in preparation,​ and login shells are available for special situations such as batch **''​wget''​** from an http server. ​ Please contact hpc-support@listserv.uark.edu if you need that.
 +
 +=== Windows ===
 +
 +Windows OS does not include Secure Copy or Secure FTP tools. ​ There are multiple file transfer clients available for download. ​ The most popular command line client is pscp.exe available here:
 +
 +https://​the.earth.li/​~sgtatham/​putty/​latest/​x86/​pscp.exe
 +
 +To transfer files using pscp.exe download and save pscp.exe to your Windows machine. ​ Then open a terminal (i.e. Command Prompt, under "​Start->​All Programs->​Accessories->​Command Prompt"​) and specify the full path to the location of the downloaded pscp.exe file followed by two arguments <​source>​ and <​destination>​. ​ Either one or both could specify a file a on remote machine (user@host:​path_to_file) or a local file (path_to_file). ​ For example:
 +
 +<​code>​
 +C:​\Users\Pawel>​ c:​\Users\Pawel\Downloads\pscp.exe filetoupload.txt pwolinsk@razor.uark.edu:​
 +</​code>​
 +
 +The code above uses secure copy protocol to upload a file "​filetoupload.txt"​ to the home directory of user pwolinsk on razor.uark.edu.
 +
 +Another popular windows transfer client (GUI) is WinSCP:
 +
 +https://​winscp.net/​eng/​download.php
 +
 +<​html>​
 +
 +<ul>
 +<table border=0><​tr><​td bgcolor=#​aaaaaa><​b>​NOTE:</​b>​ The data mover node <​b><​tt>​tgv.uark.edu</​tt></​b>​ can also be accessed as <​b><​tt>​dm.uark.edu</​tt></​b>​ from outside of the UofA network. ​ This domain name is assigned an IP address in the network DMZ (demilitarized zone) on a 10Gb/s ethernet network and is the preferred domain name to use for UofA external transfers.</​td></​tr></​table>​
 +</​html>​
  
 ===== Data Transfer between Razor & Trestles Clusters ===== ===== Data Transfer between Razor & Trestles Clusters =====
Line 30: Line 56:
 /storage /storage
 /scratch /scratch
 +There are also multiple privately owned storage areas /​storage[x].
 </​code>​ </​code>​
 When you login to **bridge**, you will be located in your Trestles **//​home//​**. ​ Also please recall that the Trestles persistent **/​scratch/​$USER** partition is being phased out and will in the future be only **/​scratch/​$PBS_JOBID** for each batch job. When you login to **bridge**, you will be located in your Trestles **//​home//​**. ​ Also please recall that the Trestles persistent **/​scratch/​$USER** partition is being phased out and will in the future be only **/​scratch/​$PBS_JOBID** for each batch job.
Line 40: Line 67:
 </​code>​ </​code>​
  
-Although **''​scp''​** to the bridge node is possible from either cluster, we recommend logging into **bridge** node directly and using the **''​cp''​** or **''​mv''​** commands to move files , thus using 40Gb/s Infiniband instead of **''​scp''​** using 1Gb/s ethernet:+Although **''​scp''​** to the bridge node is possible from either cluster, we recommend logging into **bridge** node directly and using the **''​cp''​** or **''​mv''​** commands to move files, thus using 40Gb/s Infiniband instead of **''​scp''​** using 1Gb/s ethernet:
  
 <​code>​tres-l1:​pwolinsk:​$ ssh bridge <​code>​tres-l1:​pwolinsk:​$ ssh bridge
Line 51: Line 78:
 tres-l1:​pwolinsk:​$ ​ tres-l1:​pwolinsk:​$ ​
 </​code>​ </​code>​
 +**''​rsync''​** is better for copying whole directories:​
 +<​code>​
 +bridge:​pwolinsk:​$ cd /​razor/​home/​pwolinsk
 +bridge:​pwolinsk:/​razor$ rsync -av XSEDE /​home/​pwolinsk/​
 +sending incremental file list
 +XSEDE/
 +XSEDE/​xsede13/​
 +...omitted every file listed with -v option...
 +XSEDE/​xsede14/​OpenMP_Exercise/​heat.c
 +
 +sent 185047 bytes  received 1258 bytes  124203.33 bytes/sec
 +total size is 180459 ​ speedup is 0.97
 +</​code>​
 +
 +On **bridge**, **''/​home/​username''​** and **''/​trestles/​home/​username''​** are equivalent.
 +
 **bridge** is not reachable from outside the clusters, please use **tgv**/​**dm**. ​ A data transfer node to the Trestles file systems is planned. **bridge** is not reachable from outside the clusters, please use **tgv**/​**dm**. ​ A data transfer node to the Trestles file systems is planned.
moving_data.1457557652.txt.gz · Last modified: 2016/03/09 21:07 by root