User Tools

Site Tools


spark

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
spark [2016/05/12 21:30]
pwolinsk
spark [2016/06/02 16:21]
pwolinsk
Line 1: Line 1:
 ==== Spark ==== ==== Spark ====
-Apache Spark version 1.6.1 is installed in /​share/​apps/​spark. ​+Apache Spark version 1.6.1 is installed in /​share/​apps/​spark. Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It allows users to combine the memory and cpus of multiple compute nodes into into a Spark cluster and use the aggregated cluster memory and cpus to run a single task.
  
 The example PBS script below sets up a 3 node spark cluster in standalone mode using 3 compute nodes on Trestles.  ​ The example PBS script below sets up a 3 node spark cluster in standalone mode using 3 compute nodes on Trestles.  ​
Line 39: Line 39:
 When the job starts all 3 nodes are running the worker service. ​ The job head node is also running the spark master service. ​ The log file from the spark master is saved to: When the job starts all 3 nodes are running the worker service. ​ The job head node is also running the spark master service. ​ The log file from the spark master is saved to:
  
-**''/​scratch/​$USER/​spark-$USER-org.apache.spark.deploy.master.Master-1-$HOST.out''​**+**''/​home/​$USER/​spark-$USER-org.apache.spark.deploy.master.Master-1-$HOST.out''​**
  
 and worker logs are in: and worker logs are in:
  
-**''/​scratch/​$USER/​spark-$USER-org.apache.spark.deploy.worker.Worker-<​workernum>​-$HOST.out''​**+**''/​home/​$USER/​spark-$USER-org.apache.spark.deploy.worker.Worker-<​workernum>​-$HOST.out''​**
  
 In addition to the log files a new file in the $HOME/​spark-info contains the URL of the spark master web interface: In addition to the log files a new file in the $HOME/​spark-info contains the URL of the spark master web interface:
spark.txt · Last modified: 2016/06/02 16:21 by pwolinsk