Commands to check Spark YARN Logs, Kill Spark appliation

2173

1.) To Check spark YARN logs :-
Below Command print will print out the contents of all log files from all containers of the given application.

userName@hostName:~# yarn logs -applicationId <app ID>

<app ID> is application ID provided by YARN to Spark Job, We can find out the <app ID> by looking into spark job logs or in YARN console. By Default YARN console runs on 8088 port number.

2.) To Kill Spark Job running in YARN

To kill through command prompt

userName@hostName:~# yarn application -kill <app ID>

(or)

  1. opening Spark application UI.
  2. gotojobs tab.
  3. find job among running jobs.
  4. click on kill link and confirm.

3.) To Kill Spark Job running in standalone

userName@hostName:~# spark-submit --kill [submission ID] --master [spark://...] 

userName@hostName:~# spark-submit --master spark://xxxx --kill application_xxx