User Tools

Site Tools


moving_data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
moving_data [2016/08/16 16:41]
pwolinsk
moving_data [2017/02/21 19:54] (current)
root
Line 6: Line 6:
    * **''​sftp''​** (secure ftp)  ​    * **''​sftp''​** (secure ftp)  ​
    * **''​rsync''​**    * **''​rsync''​**
 +
 +=== Linux and MacOS ===
  
 To upload a data file from the current directory on your local desktop machine to your /storage directory on **razor**: To upload a data file from the current directory on your local desktop machine to your /storage directory on **razor**:
Line 17: Line 19:
 You will also have a staging directory on **tgv**/​**dm** called **/​local_storage/​$USER/​**. ​ A new Globus Online instance on **tgv**/​**dm** is in preparation,​ and login shells are available for special situations such as batch **''​wget''​** from an http server. ​ Please contact hpc-support@listserv.uark.edu if you need that. You will also have a staging directory on **tgv**/​**dm** called **/​local_storage/​$USER/​**. ​ A new Globus Online instance on **tgv**/​**dm** is in preparation,​ and login shells are available for special situations such as batch **''​wget''​** from an http server. ​ Please contact hpc-support@listserv.uark.edu if you need that.
  
-=== Windows ​Clients ​===+=== Windows ===
  
 Windows OS does not include Secure Copy or Secure FTP tools. ​ There are multiple file transfer clients available for download. ​ The most popular command line client is pscp.exe available here: Windows OS does not include Secure Copy or Secure FTP tools. ​ There are multiple file transfer clients available for download. ​ The most popular command line client is pscp.exe available here:
Line 23: Line 25:
 https://​the.earth.li/​~sgtatham/​putty/​latest/​x86/​pscp.exe https://​the.earth.li/​~sgtatham/​putty/​latest/​x86/​pscp.exe
  
-To transfer files using pscp.exe download and save pscp.exe to your Windows machine. ​ Then open a terminal (i.e. Command Prompt, under "​Start->​All Programs->​Accessories->​Command Prompt"​) and specify the full path to the location of the downloaded pscp.exe file followed by two arguments <​source>​ and <​destination>​. ​ Either one or both could be located ​on remote ​machines ​(host:file) or local files (file). ​ For example:+To transfer files using pscp.exe download and save pscp.exe to your Windows machine. ​ Then open a terminal (i.e. Command Prompt, under "​Start->​All Programs->​Accessories->​Command Prompt"​) and specify the full path to the location of the downloaded pscp.exe file followed by two arguments <​source>​ and <​destination>​. ​ Either one or both could specify a file a on remote ​machine ​(user@host:path_to_file) or local file (path_to_file).  For example:
  
 <​code>​ <​code>​
Line 31: Line 33:
 The code above uses secure copy protocol to upload a file "​filetoupload.txt"​ to the home directory of user pwolinsk on razor.uark.edu. The code above uses secure copy protocol to upload a file "​filetoupload.txt"​ to the home directory of user pwolinsk on razor.uark.edu.
  
- +Another popular windows transfer client (GUI) is WinSCP: 
 + 
 +https://​winscp.net/​eng/​download.php 
 + 
 +<​html>​ 
 + 
 +<​ul>​ 
 +<table border=0><​tr><​td bgcolor=#​aaaaaa><​b>​NOTE:</​b>​ The data mover node <​b><​tt>​tgv.uark.edu</​tt></​b>​ can also be accessed as <​b><​tt>​dm.uark.edu</​tt></​b>​ from outside of the UofA network. ​ This domain name is assigned an IP address in the network DMZ (demilitarized zone) on a 10Gb/s ethernet network and is the preferred domain name to use for UofA external transfers.</​td></​tr></​table>​ 
 +</​html>​
  
 ===== Data Transfer between Razor & Trestles Clusters ===== ===== Data Transfer between Razor & Trestles Clusters =====
Line 46: Line 56:
 /storage /storage
 /scratch /scratch
 +There are also multiple privately owned storage areas /​storage[x].
 </​code>​ </​code>​
 When you login to **bridge**, you will be located in your Trestles **//​home//​**. ​ Also please recall that the Trestles persistent **/​scratch/​$USER** partition is being phased out and will in the future be only **/​scratch/​$PBS_JOBID** for each batch job. When you login to **bridge**, you will be located in your Trestles **//​home//​**. ​ Also please recall that the Trestles persistent **/​scratch/​$USER** partition is being phased out and will in the future be only **/​scratch/​$PBS_JOBID** for each batch job.
Line 56: Line 67:
 </​code>​ </​code>​
  
-Although **''​scp''​** to the bridge node is possible from either cluster, we recommend logging into **bridge** node directly and using the **''​cp''​** or **''​mv''​** commands to move files , thus using 40Gb/s Infiniband instead of **''​scp''​** using 1Gb/s ethernet:+Although **''​scp''​** to the bridge node is possible from either cluster, we recommend logging into **bridge** node directly and using the **''​cp''​** or **''​mv''​** commands to move files, thus using 40Gb/s Infiniband instead of **''​scp''​** using 1Gb/s ethernet:
  
 <​code>​tres-l1:​pwolinsk:​$ ssh bridge <​code>​tres-l1:​pwolinsk:​$ ssh bridge
Line 67: Line 78:
 tres-l1:​pwolinsk:​$ ​ tres-l1:​pwolinsk:​$ ​
 </​code>​ </​code>​
 +**''​rsync''​** is better for copying whole directories:​
 +<​code>​
 +bridge:​pwolinsk:​$ cd /​razor/​home/​pwolinsk
 +bridge:​pwolinsk:/​razor$ rsync -av XSEDE /​home/​pwolinsk/​
 +sending incremental file list
 +XSEDE/
 +XSEDE/​xsede13/​
 +...omitted every file listed with -v option...
 +XSEDE/​xsede14/​OpenMP_Exercise/​heat.c
 +
 +sent 185047 bytes  received 1258 bytes  124203.33 bytes/sec
 +total size is 180459 ​ speedup is 0.97
 +</​code>​
 +
 +On **bridge**, **''/​home/​username''​** and **''/​trestles/​home/​username''​** are equivalent.
 +
 **bridge** is not reachable from outside the clusters, please use **tgv**/​**dm**. ​ A data transfer node to the Trestles file systems is planned. **bridge** is not reachable from outside the clusters, please use **tgv**/​**dm**. ​ A data transfer node to the Trestles file systems is planned.
moving_data.1471365667.txt.gz · Last modified: 2016/08/16 16:41 by pwolinsk