Running a SparkSubmit Job

Prerequisites

• You have uploaded the program packages and data files required for running jobs to OBS or HDFS.
• If the job program needs to read and analyze data in the OBS file system, you need to configure storage-compute decoupling for the MRS cluster. For details, see Configuring Storage-Compute Decoupling for an MRS Cluster.

Submitting a Job on the Console

1. Log in to the MRS console.
2. On the Active Clusters page, select a running cluster and click its name to switch to the cluster details page.
3. In the Basic Information area of the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.

   Perform this step only when Kerberos authentication is enabled for the cluster.

   

   • After the IAM user synchronization is complete, wait for 5 minutes before submitting a job. For details about IAM user synchronization, see Synchronizing IAM Users to MRS..
   • When the policy of the user group an IAM user belongs to changes from MRS ReadOnlyAccess to MRS CommonOperations, MRS FullAccess, or MRS Administrator, or vice versa, it takes time for the cluster node's System Security Services Daemon (SSSD) cache to refresh. To prevent job submission failure, wait for five minutes after user synchronization is complete before submitting the job with the new policy.
   • If the IAM username contains spaces (for example, admin 01), jobs cannot be added.

4. Click Job Management. On the displayed job list page, click Create.
5. In Type, select SparkSubmit. Configure other job information.
   Figure 1 Adding a Spark job
   

   Table 1 Job configuration information

   Parameter	Description	Example
   Name	Job name. It contains 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	spark_job
   Program Path	Path of the program package to be executed. You can enter the path or click HDFS or OBS to select a file. The value contains a maximum of 1,023 characters. It cannot contain special characters (;|&>,<'$) and cannot be left blank or all spaces. The OBS program path should start with obs://, for example, obs://wordcount/program/XXX.jar. The HDFS program path should start with hdfs://, for example, hdfs://hacluster/user/XXX.jar. The SparkSubmit job execution program must end with .jar or .py.	obs://wordcount/program/test.jar
   Program Parameter	(Optional) Used to configure optimization parameters such as threads, memory, and vCPUs for the job to optimize resource usage and improve job execution performance. Table 2 lists the common program parameters of Spark jobs. You can configure the parameters based on the execution program and cluster resources.	-
   Parameters	(Optional) Key parameter for program execution. The parameter is specified by the function of the custom program. MRS is only responsible for loading the parameters. Multiple parameters are separated by spaces. The value can contain a maximum of 150,000 characters and can be left blank. The value cannot contain special characters such as ;|&><'$ CAUTION: When entering a parameter containing sensitive information (for example, login password), you can add an at sign (@) before the parameter name to encrypt the parameter value. This prevents the sensitive information from being persisted in plaintext. When you view job information on the MRS console, the sensitive information is displayed as *. Example: username=testuser @password=User password	-
   Service Parameter	(Optional) Service parameters for the job. To modify the current job, change this parameter. For permanent changes to the entire cluster, refer to Modifying the Configuration Parameters of an MRS Cluster Component and modify the cluster component parameters accordingly. Click to the add icon on the right to add more parameters. If a job needs to access OBS using AK/SK, add the following service configuration parameters: fs.obs.access.key: key ID for accessing OBS. fs.obs.secret.key: key corresponding to the key ID for accessing OBS.	-
   Command Reference	Command submitted to the background for execution when a job is submitted.	spark-submit --master yarn--deploy-mode cluster

   Table 2 Spark job running program parameters

   Parameter	Description	Example
   --conf	Configuration item for adding tasks.	spark.executor.memory=2G
   --driver-memory	Set the running memory of driver.	2G
   --num-executors	Set the number of executors to be started.	5
   --executor-cores	Set the number of executor cores.	2
   --class	Set the main class name of a task, which is specified by a function in the user program.	org.apache.spark.examples.SparkPi
   --files	Upload files to a job. The files can be user-defined configuration files or some data files from OBS or HDFS.	-
   --jars	Additional dependency JAR packages of a task, which is used to add the external dependency packages to the task.	-
   --executor-memory	Executor memory.	2G
   --conf spark-yarn.maxAppAttempts	Control the number of AM retries. If this parameter is set to 0, retry is not allowed. If this parameter is set to 1, one retry is allowed.	0

   Table 3 Job configuration information

   Parameter	Description
   Name	Job name. It contains 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed. NOTE: You are advised to set different names for different jobs.
   Program Path	Path of the program package to be executed. The following requirements must be met: The value contains a maximum of 1,023 characters. It cannot contain special characters (;|&>,<'$) and cannot be left blank or all spaces. The path of the program to be executed can be stored in HDFS or OBS. The path varies depending on the file system. OBS: The path must start with s3a://. Example: s3a://wordcount/program/xxx.jar HDFS: The path must start with /user. For details about how to import data to HDFS, see Uploading Application Data to an MRS Cluster. For SparkScript, the path must end with .sql. For MapReduce and Spark, the path must end with .jar. The .sql and .jar are case-insensitive.
   Parameters	Key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter. Multiple parameters are separated by space. Configuration method: Package name.Class name The parameter contains a maximum of 150,000 characters. It cannot contain special characters ;|&><'$, but can be left blank. CAUTION: When entering a parameter containing sensitive information (for example, login password), you can add an at sign (@) before the parameter name to encrypt the parameter value. This prevents the sensitive information from being persisted in plaintext. When you view job information on the MRS console, the sensitive information is displayed as *. Example: username=testuser @password=User password
   Import From	Path for inputting data Data can be stored in HDFS or OBS. The path varies depending on the file system. OBS: The path must start with s3a://. HDFS: The path must start with /user. For details about how to import data to HDFS, see Uploading Application Data to an MRS Cluster. The value contains a maximum of 1,023 characters and cannot contain special characters (;|&>,<'$). This parameter can be left blank.
   Export To	Path for outputting data NOTE: When setting this parameter, select OBS or HDFS. Select a file directory or manually enter a file directory, and click OK. If you add the hadoop-mapreduce-examples-x.x.x.jar sample program or a program similar to hadoop-mapreduce-examples-x.x.x.jar, enter a directory that does not exist. Data can be stored in HDFS or OBS. The path varies depending on the file system. OBS: The path must start with s3a://. HDFS: The path must start with /user. The value contains a maximum of 1,023 characters and cannot contain special characters (;|&>,<'$). This parameter can be left blank.
   Log Path	Path for storing job logs that record job running status. Data can be stored in HDFS or OBS. The path varies depending on the file system. OBS: The path must start with s3a://. HDFS: The path must start with /user. The value contains a maximum of 1,023 characters and cannot contain special characters (;|&>,<'$). This parameter can be left blank.

6. Confirm job configuration information and click OK.
7. After the job is submitted, you can view the job running status and execution result in the job list. After the job status changes to Completed, you can view the analysis result of related programs.

Submitting a Job Using the Cluster Client

1. Install the MRS cluster client. For details, see Installing an MRS Cluster Client.

   The MRS cluster comes with a client installed for job submission by default, which can also be used directly. For MRS 3.x and later versions, the default client installation path is /opt/Bigdata/client on the Master node. For versions earlier than MRS 3.x, the default client installation path is /opt/client on the Master node.

2. Upload the application to be run to the node where the cluster client is located.
   

   • The JAR sample program used in this section is {Cluster client installation directory}/Spark2x/spark/examples/jars/spark-examples_*.jar. (In some versions, the name of the Spark2x folder in the cluster is Spark. Replace it with the actual name.)
   • You can log in to the client node and run the following command to upload the sample JAR package to be executed to the HDFS.

     For example, upload the file to the /tmp directory of HDFS.

     hdfs dfs -put {Client installation directory}/Spark2x/spark/examples/jars/spark-examples_*.jar /tmp

3. If Kerberos authentication has been enabled for the current cluster, create a user for submitting jobs by referring to Creating an MRS Cluster User.

   Skip this step for normal clusters.

   In this example, a machine-machine user has been created, and user groups (hadoop and supergroup), the primary group (supergroup), and role permissions (System_administrator and default) have been correctly assigned to the user.

   After the user is created, download the authentication credential file.

   • For clusters of MRS 3.x or later, log in to FusionInsight Manager and choose System > Permission > User. In the Operation column of the newly created user, choose More > Download Authentication Credential.
   • For MRS 2.x or earlier, log in to MRS Manager and choose System > Manage User. In the Operation column of the newly created user, choose More > Download Authentication Credential.

   Upload the user authentication credential to the /opt directory on the cluster client node.

4. Log in to the node where the client is located as the MRS cluster client installation user.
5. Decompress the user authentication credential file to obtain the user.keytab and krb5.conf files.

   cd /opt

   tar -xvf XXX_keytab.tar

6. Initialize environment variables.

   cd /opt/Bigdata/client

   source bigdata_env

   cd $SPARK_HOME

7. Run the following command to submit a Spark job:

   ./bin/spark-submit --master yarn --deploy-mode client --conf spark.yarn.principal=MRSTest --conf spark.yarn.keytab=/opt/user.keytab --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_*.jar 10

   ...
   Pi is roughly 3.1402231402231404

   Parameter description:

   • deploy-mode: running mode of the Spark driver. The value can be client or cluster. In this example, the client mode is used to submit the job.
   • conf: additional configuration attribute. For example, if the keytab file is used for user authentication in this example, configure the following parameters:
     • spark.yarn.principal: name of the user who submits the job.
     • spark.yarn.keytab: keytab file for user authentication.
   • class: main class name of the application, which is specified by the running application.
   • XXX.jar: program run by the job