使用SparkSubmitOperator提交作业，该作业使用ssh作为conn_type

Question

我在交流Spark和Airflow时遇到问题。尽管我可以使用ssh连接运行我的spark作业，但是我需要使用SparkSubmitOperator从Airflow提交作业。在我的气流机器中，没有与火花相关的东西，例如火花提交。我的集群是使用docker-swarm构建的。 Dag文件如下：

from airflow import DAG
from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
from datetime import datetime, timedelta
import os
from os import path
import datetime

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.datetime(2018, 4, 11),
    'retries': 0,
    'retry_delay': datetime.timedelta(minutes=5),
    'depends_on_past': False
}

dag = DAG('sparkjob', default_args=default_args, schedule_interval=timedelta(1))

operator = SparkSubmitOperator(
    task_id='spark_submit_job',
    conn_id='spark_default',
    java_class='Main',
    application='/usr/local/airflow/SimpleSpark.jar',
    verbose=False,
    dag=dag
)

此外，配置屏幕如下：

此配置是由于以下错误产生的：

[2018-10-05 08:08:38,819] {{logging_mixin.py:95}} INFO - [2018-10-05 08:08:38,818] {{base_hook.py:83}} INFO - Using connection to: spark://spark-master:7077

[2018-10-05 08:08:38,822] {{logging_mixin.py:95}} INFO - [2018-10-05 08:08:38,821] {{spark_submit_hook.py:283}} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'spark://spark-master:7077', '--name', 'airflow-spark', '--class', 'Main', '--queue', 'root.default', '/usr/local/airflow/SimpleSpark.jar']

[2018-10-05 08:08:38,824] {{models.py:1736}} ERROR - [Errno 2] No such file or directory: 'spark-submit': 'spark-submit'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 168, in execute
    self._hook.submit(self._application)
  File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 330, in submit
    **kwargs)
  File "/usr/local/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1344, in _execute_child

我认为，我无法执行ssh连接。因为，我没有指定remote_host，也不知道在哪里标识。