Question

我正在为html_safe用例探索argo workflow。是否有示例spark展示了如何使用 Argo工作流程

在YAML上执行spark job

Answer 1

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-spark-
spec:
  entrypoint: sparkapp
  templates:
  - name: sparkapp
  container:
    image: sparkimage
    command: [sh]
    args: [
                    "-c",
                    "sh /opt/spark/bin/spark-submit.sh  \"--class\" \"org.apache.spark.examples.SparkPi\" \"/opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar\" "
                ]

希望这会有所帮助！

Answer 2

这里是运行Spark的Pi示例的示例，只需替换k8s api的图像，类和url的正确值

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: wf-spark-pi
  namespace: spark
spec:
  entrypoint: sparkapp
  templates:
  - name: sparkapp
    container:
      image: Spark-Image
      imagePullPolicy: Always
      command: [sh]
      args:
      - /opt/spark/bin/spark-submit 
      - --master 
      - k8s://https://<K8S_API_TCP_ADDR>:<K8S_API_TCP_PORT>
      - --deploy-mode
      - cluster
      - --conf 
      - spark.kubernetes.namespace=spark
      - --conf
      - spark.kubernetes.container.image=Spark-Image
      - --conf
      - spark.kubernetes.driver.pod.name=spark
      - --conf 
      - spark.executor.instances=2
      - --class
      - org.apache.spark.examples.SparkPi
      - local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar
      resources: {}      
    restartPolicy: OnFailure

使用Argo Workflow的K8S Spark

2 个答案: