Skip to content

Instantly share code, notes, and snippets.

@tmusabbir

tmusabbir/livy-example.sh

Created March 27, 2018 22:41
Show Gist options
  • Save tmusabbir/936d0ad13852a1d1fc945b869debb31e to your computer and use it in GitHub Desktop.
Save tmusabbir/936d0ad13852a1d1fc945b869debb31e to your computer and use it in GitHub Desktop.
Download ZIP
Sample commands for spark-submit using Apache Livy
Raw
livy-example.sh
# This is the usual sample spark-submit command to submit the SparkPi sample application
spark-submit --class org.apache.spark.examples.SparkPi /usr/lib/spark/examples/jars/spark-examples.jar
# Now submit the same job from EMR master node (assume the jar file is in test folder):
curl -X POST --data '{"file": "/test/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" localhost:8998/batches
# Previous example is pointing to localhost as it submitted job from the same host, now submitting job from remote location:
curl -X POST --data '{"file": "/test/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" <<your-emr-master-dns>>:8998/batches
# Now assume the jar file is in S3 location, in that case, you can follow this:
curl -X POST --data '{"file": "s3://<<your-bucket>>/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" <<your-emr-master-dns>>:8998/batches
# Check the status of Spark jobs:
curl <<your-emr-master-dns>>:8998/sessions/0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment