Question

我有一个Airflow DAG，用于提交火花作业，为此我使用了SparkSubmitOperator。在DAG中，我必须指定需要运行的应用程序JAR。目前，它被硬编码为spark-job-1.0.jar，如下所示：

from airflow import DAG

from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
from datetime import datetime, timedelta


args = {
    'owner': 'joe',
    'start_date': datetime(2020, 5, 23)
}
dag = DAG('spark_job', default_args=args)

operator = SparkSubmitOperator(
    task_id='spark_sm_job',
    conn_id='spark_submit',
    java_class='com.mypackage',
    application='/home/ubuntu/spark-job-1.0.jar',       <---------
    total_executor_cores='1',
    executor_cores='1',
    executor_memory='500M',
    num_executors='1',
    name='airflow-spark-job',
    verbose=False,
    driver_memory='500M',
    application_args=["yarn", "10.11.21.12:9092"],
    dag=dag,
)

问题是，发行版名称会增加，我尝试使用通配符spark-job-*.jar，但没有用！ -是否可以使用通配符，或者还有其他解决方法？

Answer 1

假设始终只有一个文件匹配'use strict' let fontOverwrite = 'h1 {font-family: Comic Sans MS !important}' let transformOverwrite = 'h1 {text-transform:uppercase}' // if we don't have the styleDom, create it let styleDom = document.getElementById('comic-sans-everything-style') if (!styleDom) { styleDom = document.querySelector('h1').appendChild(document.createElement('style')) styleDom.id = 'comic-sans-everything-style' styleDom.rel = 'stylesheet' styleDom.type = 'text/css' } const updateStyle = () => { window.chrome.storage.sync.get(['status', 'uppercase'], (items) => { if (!window.chrome.runtime.error) { if (items.status && items.uppercase) { styleDom.innerText = fontOverwrite + transformOverwrite } else if (items.status) { styleDom.innerText = fontOverwrite } else { styleDom.innerText = '' } } }) } // ADDED to make letters rainbow :) function changeColor() { var paragraphs = document.getElementsByTagName("h1"); for(var i = 0; i < paragraphs.length; i++) { var innerText = paragraphs[i].innerHTML; var innerTextSplit = innerText.split(""); paragraphs[i].innerText = ""; var isHTMLElement = false; for(var j = 0; j < innerTextSplit.length; j++) { if(innerTextSplit[j] == "<") isHTMLElement = true; if(!isHTMLElement){ var randomColor = "rgb(" + Math.floor((Math.random() * 255) + 1) + ", " + Math.floor((Math.random() * 255) + 1) + ", " + Math.floor((Math.random() * 255) + 1) + ");" innerTextSplit[j] = '<span style="color: ' + randomColor + '">' + innerTextSplit[j] + '</span>'; } if(innerTextSplit[j] == ">") isHTMLElement = false; } innerTextSplit = innerTextSplit.join(''); paragraphs[i].innerHTML += innerTextSplit; } } //THIS is where i'm trying to make the functions happen, when they're both there ONLY changeColor() works // run once on file load updateStyle() changeColor()，那么您可以执行以下操作：

/home/ubuntu/spark-job-*.jar

SparkSubmitOperator Airflow DAG中的应用程序JAR名称是否可以使用通配符？

1 个答案: