airflow.exceptions.AirflowException:使用密钥文件

时间:2017-06-14 06:46:46

标签: python python-2.7 airflow dataflow

问题:在使用airflow运行数据流作业时,我遇到错误,是否与我们提供的关键路径有关。如果是,那么正确的道路应该是什么。错误:

File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1374, in run 

result = task_copy.execute(context=context) File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/dataflow_operator.py", line 114, in execute
self.jar = bucket_helper.google_cloud_to_local(self.jar)

File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/dataflow_operator.py", line 229, in google_cloud_to_local
file_size = self._gcs_hook.download(bucket_id, object_id, local_file)
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcs_hook.py", line 58, in download
service = self.get_conn()
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcs_hook.py", line 43, in get_conn
http_authorized = self._authorize()
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcp_api_base_hook.py", line 84, in _authorize
raise AirflowException('Scope should be defined when using a key file.')
AirflowException: Scope should be defined when using a key file.
[2017-06-13 20:05:45,165] {models.py:1433} INFO - Marking task as UP_FOR_RETRY
[2017-06-13 20:05:45,174] {models.py:1462} ERROR - Scope should be defined when using a key file.
Traceback (most recent call last):
File "/usr/bin/airflow", line 28, in <module>
args.func(args)
File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 585, in test
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
result = task_copy.execute(context=context)
File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/dataflow_operator.py", line 114, in execute
self.jar = bucket_helper.google_cloud_to_local(self.jar)
File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/dataflow_operator.py", line 229, in google_cloud_to_local
file_size = self._gcs_hook.download(bucket_id, object_id, local_file)
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcs_hook.py", line 58, in download
service = self.get_conn()
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcs_hook.py", line 43, in get_conn
http_authorized = self._authorize()
File "/usr/lib/python2.7/site-packages/airflow/contrib/hooks/gcp_api_base_hook.py", line 84, in _authorize
raise AirflowException('Scope should be defined when using a key file.')
airflow.exceptions.AirflowException: Scope should be defined when using a key file.

有人可以帮我吗?。提前谢谢

1 个答案:

答案 0 :(得分:1)

您应在Airflow中指定连接的“范围”字段。在网络用户界面中,转到顶部菜单中的Admin > Connections,然后点击google_cloud_default的修改链接。对于GCS,可以找到范围列表here。最常用的是https://www.googleapis.com/auth/cloud-platform