Question

使用EmrAddStepsOperator时，Airflow任务执行出现错误。

我已经能够使用EmrCreateJobFlowOperator创建集群。但是我无法向集群添加步骤。我的代码在旧版本的Airflow中似乎可以正常工作。我正在从1.8.0升级到1.10.3。我也从python2更改为python3。

这是我在任务日志中看到的：

[2019-06-18 08:28:52,411] {__init__.py:1374} INFO - Executing <Task(EmrAddStepsOperator): car_global_it> on 2019-06-18T06:28:33.490242+00:00
[2019-06-18 08:28:52,411] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'CARS', 'car_global_it', '2019-06-18T06:28:33.490242+00:00', '--job_id', '194', '--raw', '-sd', 'DAGS_FOLDER/afcontrol/cars.py', '--cfg_path', '/tmp/tmp9939gbb5']
Traceback (most recent call last):
  File "/home/airflow/.virtualenvs/airflow/bin/airflow", line 21, in <module>
    from airflow import configuration
  File "/home/airflow/.virtualenvs/airflow/lib64/python3.7/site-packages/airflow/__init__.py", line 38, in <module>
    from airflow import settings, configuration as conf
  File "/home/airflow/.virtualenvs/airflow/lib64/python3.7/site-packages/airflow/settings.py", line 33, in <module>
    from sqlalchemy.orm import scoped_session, sessionmaker
  File "/home/airflow/.virtualenvs/airflow/lib64/python3.7/site-packages/sqlalchemy/orm/__init__.py", line 291, in <module>
    __go(locals())
  File "/home/airflow/.virtualenvs/airflow/lib64/python3.7/site-packages/sqlalchemy/orm/__init__.py", line 277, in __go
    from . import events  # noqa
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 818, in get_code
  File "<frozen importlib._bootstrap_external>", line 917, in get_data
MemoryError

{jobs.py:2562} INFO - Task exited with return code 1

我认为我的代码在新版本中应该可以正常运行。日志并没有告诉我太多。还有其他方法可以对Airflow进行更深入的调试吗？知道会是什么吗？

谢谢。

运行EmrAddStepsOperator时气流中出现MemoryError

0 个答案: