Question

我正在尝试确定可能导致以下问题（气流）的原因？

基本上我已经编写了一个Test DAG，它的主要任务是从BigQuery读取数据并将其写入新表中。我尝试搜索此问题，但我无法找出可能导致此问题的原因。我什至不确定我的gcp_connection是否正常工作。我不知道该如何测试。任何帮助将不胜感激！

Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/airflow/models.py", line 1659, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/anaconda3/lib/python3.6/site-packages/airflow/operators/subdag_operator.py", line 103, in execute
    executor=self.executor)
  File "/anaconda3/lib/python3.6/site-packages/airflow/models.py", line 4214, in run
    job.run()
  File "/anaconda3/lib/python3.6/site-packages/airflow/jobs.py", line 203, in run
    self._execute()
  File "/anaconda3/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/anaconda3/lib/python3.6/site-packages/airflow/jobs.py", line 2547, in _execute
    raise AirflowException(err)
airflow.exceptions.AirflowException: ---------------------------------------------------
Some task instances failed:
{('test_oscope.test_oscope', 'create_if_not_exists', datetime.datetime(2016, 6, 1, 0, 0, tzinfo=<Timezone [UTC]>), 1), ('test_oscope.test_oscope', 'fill', datetime.datetime(2016, 6, 1, 0, 0, tzinfo=<Timezone [UTC]>), 1)}

Answer 1

气流挂钩本身非常有用，可用于交互式环境（如iPython或Jupyter Notebook）。

例如：

from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook

GCSHook = GoogleCloudStorageHook(google_cloud_storage_conn_id='google_conn_id')
GCSHook.get_conn() # This will check if your GCP connection is working correctly or not

无法调试气流错误。尝试使用气流在GCP上建立数据管道

我正在尝试确定可能导致以下问题（气流）的原因？

1 个答案: