Question

我对Apache Airflow不满意。从字面上看才刚刚开始，我遇到了一个错误。我写了我的第一个数据，并且正在调用Python脚本。当我设置它时，它成功运行并开始工作，并且我计划每天运行一次。我今天来检查它，但dag失败并显示错误消息-HTTP错误404：找不到。

对我来说，一切都是新奇的，因此很抱歉，如果这很容易解决，但是我不明白为什么会出现404错误。我尝试重新启动docker，看看这是否是网络服务器问题，但没有运气。

感谢您的帮助

Screenshot

DAG

from airflow.models import DAG
from datetime import datetime, timedelta
from airflow.operators.python_operator import PythonOperator
from covid_cases import covid_data


default_args = {
    'owner': 'airflow',
    'start_date': datetime(2020, 10, 4),
    'retries': 2,
    'retry_delay': timedelta(seconds=20)}

dag =  DAG(dag_id = 'covid_updates',
           default_args = default_args,
           schedule_interval = "0 4 * * *")

t1 = PythonOperator(task_id = 'covid_update',
                    python_callable = covid_data,
                    dag = dag)

t1

PythonOperator-covid_cases.py

def covid_data():
    """

    Overview:
    ---------
    Downloads the USA COVID data directly from John Hopkins CSSEGISandData.
    This function merges all data from most current date to earliest date (2020-4-11).
    Using this function a user can conduct time series analysis in how COVID
    increases/decreases in various states.

    Output:
    -------
    One uncleaned .csv file called "usa_covid_cases.csv"


    """
    from datetime import datetime, timedelta
    import pandas as pd
    from urllib.error import HTTPError

    # Set starting index
    i = 1

    # Earliest dataset available on GitHub
    start_date = datetime.strptime('2020-4-11', '%Y-%m-%d').date()

    # Pulling today's date minus 1 day due to delay posting on GitHub
    today = datetime.now().date() - timedelta(days=i)

    # Setting llist to store dataframe file names
    file_names = []

    # Looping until date is equal to earlist date = Start Date
    while not (start_date.day == today.day and start_date.month == today.month and start_date.year == today.year):

        # Extracting variables from current date
        day = today.day
        month = today.month
        year = today.year

        # Cleaning and converting values for formatting on GitHub URL link
        if day < 10:
            day = '0' + str(day)

        if month < 10:
            month = '0' + str(month)

        # Setting variable for each url
        url = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports_us/{}-{}-{}.csv'\
                    .format(month, day, str(year))

        try:
            # Reading each url as a datafra,
            df = pd.read_csv(url, error_bad_lines=False)
        except HTTPError as e:
            # handle the error (print, log, etc)
            continue
        finally:
            # Code moved here to prevent an endless loop
            # Subtracting the new index to increase 1 less day from the current date
            today = datetime.now().date() - timedelta(days=i)

        # Saving each dataframe into the empty list
        file_names.append(df)

        # Increasing the index by 1
        i += 1

    # Once while loop ends - concat all the files into a single dataframe
    new_df = pd.concat(file_names)

    # Save output into new csv file
    new_df.to_csv('usa_covid_cases.csv')

Answer 1

从日志中，当您尝试从URL读取数据并且该URL不存在时会发生错误。

pd.read_csv(url, error_bad_lines=False) #Line 50 of covid_data.py

Answer 2

我刚刚看过GitHub repo，最早的数据集来自 2020年12月12日。

为防止DAG无法从可能的已删除数据集中失败，您可以将pd.read_csv()包装在try except块中，如下所示：

# Import the HTTPError (error from you screenshot)
from urllib.error import HTTPError
...

try:
    # Reading each url as a datafra,
    df = pd.read_csv(url, error_bad_lines=False)
except HTTPError as e:
    # handle the error (print, log, etc)
    continue
finally:
    # Code moved here to prevent an endless loop
    # Subtracting the new index to increase 1 less day from the current date
    today = datetime.now().date() - timedelta(days=i)

 # Saving each dataframe into the empty list
 file_names.append(df)

 # Increasing the index by 1
 i += 1

...

气流DAG失败-错误-HTTP错误404：找不到

DAG

PythonOperator-covid_cases.py

2 个答案: