并行运行22个气流工作人员吊舱时，k8s集群挂起

时间：2019-08-15 10:52:01

标签： mysql kubernetes airflow airflow-scheduler docker-desktop

我正在docker-desktop kubernetes集群（在Mac上）上使用Kubernetes executor运行气流。我在dag文件中有多个sensorOperators，每个传感器都是下游依赖项的一部分。在总共22个传感器操作员并行运行的情况下，执行5-7分钟后，我的kubernetes群集连接断开。重新启动集群后，我可以再次访问我的k8s dashbaord并检查所有red失败的任务的日志，它们似乎抱怨mysql连接失败。 image 

[2019-08-15 10:09:03,829] {__init__.py:1374} INFO - Executing <Task(IngestionStatusSensor): ingestion_ready_relational_character_creation> on 2019-03-15T00:00:00+00:00
[2019-08-15 10:09:03,829] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'datascience_ecc_v1', 'ingestion_ready_relational_character_creation', '2019-03-15T00:00:00+00:00', '--job_id', '22', '--raw', '-sd', 'DAGS_FOLDER/DAG_datascience_ecc_v1.py', '--cfg_path', '/tmp/tmpb3993h8h']
[2019-08-15 10:10:00,468] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation [2019-08-15 10:10:00,447] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=10, pool_recycle=1800, pid=11
[2019-08-15 10:12:39,448] {logging_mixin.py:95} INFO - [2019-08-15 10:12:39,381] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:42,967] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation [2019-08-15 10:12:42,772] {__init__.py:51} INFO - Using executor LocalExecutor
[2019-08-15 10:12:44,651] {logging_mixin.py:95} INFO - [2019-08-15 10:12:44,651] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:45,331] {logging_mixin.py:95} INFO - [2019-08-15 10:12:45,331] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:45,364] {logging_mixin.py:95} INFO - [2019-08-15 10:12:45,364] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:50,394] {logging_mixin.py:95} INFO - [2019-08-15 10:12:50,394] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:55,415] {logging_mixin.py:95} INFO - [2019-08-15 10:12:55,415] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:55,529] {logging_mixin.py:95} INFO - [2019-08-15 10:12:55,528] {jobs.py:195} ERROR - Scheduler heartbeat got an exception: (_mysql_exceptions.OperationalError) (2006, "Unknown MySQL server host 'mysql' (111)") (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-08-15 10:12:58,758] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation [2019-08-15 10:12:58,724] {cli_action_loggers.py:70} ERROR - Failed on pre-execution callback using <function default_action_log at 0x7f7452d13730>
[2019-08-15 10:12:58,758] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation Traceback (most recent call last):
[2019-08-15 10:12:58,759] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect
[2019-08-15 10:12:58,759] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return fn()
[2019-08-15 10:12:58,759] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 434, in connect
[2019-08-15 10:12:58,759] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return _ConnectionFairy._checkout(self)
[2019-08-15 10:12:58,775] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 831, in _checkout
[2019-08-15 10:12:58,775] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     fairy = _ConnectionRecord.checkout(pool)
[2019-08-15 10:12:58,775] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 563, in checkout
[2019-08-15 10:12:58,775] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     rec = pool._do_get()
[2019-08-15 10:12:58,775] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1259, in _do_get
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     self._dec_overflow()
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     compat.reraise(exc_type, exc_value, exc_tb)
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     raise value
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1256, in _do_get
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return self._create_connection()
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 379, in _create_connection
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return _ConnectionRecord(self)
[2019-08-15 10:12:58,776] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 508, in __init__
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     self.__connect(first_connect_check=True)
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/pool.py", line 710, in __connect
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     connection = pool._invoke_creator(self)
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return dialect.connect(*cargs, **cparams)
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 437, in connect
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return self.dbapi.connect(*cargs, **cparams)
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     return Connection(*args, **kwargs)
[2019-08-15 10:12:58,777] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation   File "/usr/local/airflow/venv/lib/python3.6/site-packages/MySQLdb/connections.py", line 208, in __init__
[2019-08-15 10:12:58,778] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation     super(Connection, self).__init__(*args, **kwargs2)
[2019-08-15 10:12:58,778] {base_task_runner.py:101} INFO - Job 22: Subtask ingestion_ready_relational_character_creation _mysql_exceptions.OperationalError: (2006, "Unknown MySQL server host 'mysql' (111)")

但是，如果我从气流UI仪表板禁用了dag，并独立运行每个失败的任务，它们似乎可以成功运行。我以为可能对mysql有连接限制，所以我在airflow core configs中添加了以下内容

sql_alchemy_pool_enabled=True
sql_alchemy_pool_size = 10
sql_alchemy_max_overflow = 15
sql_alchemy_pool_recycle = 1800
sql_alchemy_reconnect_timeout = 300

我还尝试将airconfig.cfg中的parallelism和dag_concurrency分别增加到32和40。但是这两个配置都没有任何效果。我不知道是什么导致了这种失败。要么集群先关闭，然后工作吊舱无法与mysql服务器连接，否则将无法进行连接。 docker-desktop kubernetes集群是否存在问题？我应该看看kube-dns的日志吗？

更新 在我一起执行3个dag任务之后，群集再次挂起，这一次，airflow-webserver也放弃了

Traceback (most recent call last):
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 397, in _error_catcher
    yield
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 704, in read_chunked
    self._update_chunk_length()
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 643, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 293, in run
    self.worker_uuid)
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 314, in _run
    **kwargs):
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/kubernetes/watch/watch.py", line 144, in stream
    for line in iter_resp_lines(resp):
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/kubernetes/watch/watch.py", line 48, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 732, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 415, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[2019-08-13 14:39:03,684] {kubernetes_executor.py:295} ERROR - Unknown error in KubernetesJobWatcher. Failing
Traceback (most recent call last):
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 639, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
  ...

    File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/usr/local/airflow/venv/lib/python3.6/site-packages/urllib3/response.py", line 415, in _error_catcher
        raise ProtocolError('Connection broken: %r' % e, e)
    urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
    ...
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/airflow/venv/lib/python3.6/site-packages/MySQLdb/connections.py", line 208, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2006, "Unknown MySQL server host 'mysql' (111)")

我不太想知道在哪里寻找信息，但是如果有人告诉我，我会根据需要提供相关日志。

环境： 码头工人：19.03.1， kubernetes：1.14.3， 气流：1.10.3 mysql：5.7，

2 个答案:

答案 0 :(得分：1)

似乎是资源问题。

您可以尝试在Docker设置中增加CPU和内存，并连接到Docker VM以检查资源使用情况。

或者，您可以在Virtualbox中创建一个具有一或两个节点的主群集，然后尝试在其中运行任务。在这种情况下，主节点将不会遭受资源短缺的困扰，并且群集仍应可用于连接。

Here是如何连接到Mac的docker-desktop

请告诉我是否有帮助。

答案 1 :(得分：1)

Airflow的默认任务并发性设置为32，然后尝试同时运行所有22个Pod，所需的内存比我的k8s集群所能承受的更多。将气流配置中的AIRFLOW__CORE__PARALLELISM从32（默认值）减少到8可以为我完成工作。当然，对于那些需要并行运行更多任务的业务案例的人，可以选择增加内存和多集群。

相关问题
最新问题