Question

我遇到错误

黄昏-警告-工作者超出了95％的内存预算。

我正在使用具有4个物理核和8个虚拟核的本地PC进行尝试：

每...

Managing worker memory on a dask localcluster

...以及此处的文档...

https://distributed.readthedocs.io/en/latest/worker.html#memory-management

...我尝试编辑.config \ dask \ distributed.yaml以取消对底部五行的注释...

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 0.60  # target fraction to stay below
      spill: 0.70  # fraction at which we spill to disk
      pause: 0.80  # fraction at which we pause worker threads
      terminate: 0.95  # fraction at which we terminate the worker

我也在代码中尝试了以下方法：

from dask.distributed import Client, LocalCluster

    worker_kwargs = {
        'memory_limit': '1G',
        'memory_target_fraction': 0.6,
        'memory_spill_fraction': 0.7,
        'memory_pause_fraction': 0.8,
    #     'memory_terminate_fraction': 0.95,
    }

    cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)
    client = Client(cluster, memory_limit='4GB')

...带有和不带有Client（）函数的memory_limit参数。

有什么想法吗？

Answer 1

如果您不希望dask终止工作程序，则除了取消注释之外，还需要进行以下更改：

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 0.60  # target fraction to stay below
      spill: 0.70  # fraction at which we spill to disk
      pause: 0.80  # fraction at which we pause worker threads
      terminate: False  # fraction at which we terminate the worker

Dask-警告-工作者超出了95％的内存预算

1 个答案: