Question

我正在尝试使用dask-yarn在集群上分发Python作业。

我正在使用以下代码创建集群：

from dask_yarn import YarnCluster

cluster = YarnCluster(environment='.conda/envs/myconda', worker_vcores=2, worker_memory='4GB', n_workers=4)
client = Client(cluster)
cluster.shutdown()

我本以为您需要更多信息以确保它已连接到纱线上。

错误消息如下所示：

---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-16-9de74c663703> in <module>()
----> 1 cluster = YarnCluster('myconda.tar.gz')

~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in __init__(self, environment, n_workers, worker_vcores, worker_memory, worker_restarts, worker_env, scheduler_vcores, scheduler_memory, deploy_mode, name, queue, tags, user, skein_client)
    293                                    user=user)
    294 
--> 295         self._start_cluster(spec, skein_client)
    296 
    297     @cached_property

~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in _start_cluster(self, spec, skein_client)
    373             app = skein_client.submit_and_connect(spec)
    374             try:
--> 375                 scheduler_address = app.kv.wait('dask.scheduler').decode()
    376                 dashboard_address = app.kv.get('dask.dashboard')
    377                 if dashboard_address is not None:

~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in wait(self, key, return_owner)
    653                 return res
    654 
--> 655             event = event_queue.get()
    656 
    657         return event.result if return_owner else event.result.value

~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in get(self, block, timeout)
    279         if isinstance(out, Exception):
    280             self._exception = out
--> 281             raise out
    282         return out
    283 

ConnectionError: Unable to connect to application
```python

Answer 1

这可能是因为您的应用程序无法启动（因此无法连接）。我建议查看日志：

$ yarn logs -applicationId <YOUR APPLICATION ID>

从上面的代码看，奇怪的是指定的conda环境。看来您正在传递目录的路径，而不是在已归档的环境中传递-您可能想在这里阅读文档：http://yarn.dask.org/en/latest/environments.html。

群集上的dask-yarn：无法连接到应用程序

1 个答案: