我一直在尝试为Preprocessing和随后的Feature selection和Machine Learning算法编写自定义类。
我使用(preprocessing only)破解了@delayed。但是，当我从tutorials中读取信息时，使用Client可以达到相同的目的。它造成了两个问题。
作为脚本运行。不像Jupyter笔记本
第一个问题：
# Haven't run any scheduler or worker manually
client = Client() # Nothing passed as an argument
# Local Cluster is not working;
Error:...
if __name__=='__main__':
freeze_support()
...
我在Jupyter Notebook中尝试了相同的方法，但是没有在不同的终端上运行任何调度程序或工作程序。 有效！
现在，我用1个调度程序和2个工作程序触发了3个终端，并在脚本中将其更改为Client('IP')。 已解决错误，此行为的任何原因。
第二个问题：
问题标题中提到的错误。将client = Client('IP')作为参数传递给构造函数，并将self.client.submit事物传递给集群。但是失败，并显示错误消息
错误：没有模块名称'diya_info'
代码如下：
main.py
import dask.dataframe as dd
from diya_info import Diya_Info
import time
# from dask import delayed
from dask.distributed import Client
df = dd.read_csv(
'/Users/asifali/workspace/playground/flask/yellow_tripdata_2015- 01.csv')
# df = delayed(df.fillna(0.3))
# df = df.compute()
client = Client('192.168.0.129:8786')
X = df.drop('payment_type', axis=1).copy()
y = df['payment_type']
Instance = Diya_Info(X, y, client)
s = time.ctime(int(time.time()))
print(s)
Instance = Instance.fit(X, y)
e = time.ctime(int(time.time()))
print(e)
# print((e-s) % 60, ' secs')
diya_info.py
from sklearn.base import TransformerMixin, BaseEstimator
from dask.multiprocessing import get
from dask import delayed, compute
class Diya_Info(BaseEstimator, TransformerMixin):
def __init__(self, X, y, client):
assert X is not None, 'X can\'t be None'
assert type(X).__name__ == 'DataFrame', 'X not of type DataFrame'
assert y is not None, 'y can\'t be None'
assert type(y).__name__ == 'Series', 'y not of type Series'
self.client = client
def fit(self, X, y):
self.X = X
self.y = y
# X_status = self.has_null(self.X)
# y_status = self.has_null(self.y)
# X_len = self.get_len(self.X)
# y_len = self.get_len(self.y)
X_status = self.client.submit(self.has_null, self.X)
y_status = self.client.submit(self.has_null, self.y)
X_len = self.client.submit(self.get_len, self.X)
y_len = self.client.submit(self.get_len, self.y)
# X_null, y_null, X_length, y_length
X_null, y_null, X_length, y_length = self.client.gather(
[X_status, y_status, X_len, y_len])
assert X_null == False, 'X contains some columns with null/NaN values'
assert y_null == False, 'y contains some columns with null/NaN values'
assert X_length == y_length, 'Shape mismatch, X and y are of different length'
return self
def transform(self, X):
return X
@staticmethod
# @delayed
def has_null(df):
return df.isnull().values.any()
@staticmethod
# @delayed
def get_len(df):
return len(df)
这是完整的堆栈跟踪：
Sat Aug 11 13:29:08 2018
distributed.utils - ERROR - No module named 'diya_info'
Traceback (most recent call last):
File "/anaconda3/lib/python3.6/site-packages/distributed/utils.py", line 238, in f
result[0] = yield make_coro()
File "/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/anaconda3/lib/python3.6/site-packages/distributed/client.py", line 1315, in _gather
traceback)
File "/anaconda3/lib/python3.6/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/anaconda3/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 59, in loads
return pickle.loads(x)
ModuleNotFoundError: No module named 'diya_info'
Traceback (most recent call last):
File "notebook/main.py", line 24, in <module>
Instance = Instance.fit(X, y)
File "/Users/asifali/workspace/pythonProjects/ML-engine-DataX/pre-processing/notebook/diya_info.py", line 28, in fit
X_status, y_status, X_len, y_len)
File "/anaconda3/lib/python3.6/site-packages/distributed/client.py", line 2170, in compute
result = self.gather(futures)
File "/anaconda3/lib/python3.6/site-packages/distributed/client.py", line 1437, in gather
asynchronous=asynchronous)
File "/anaconda3/lib/python3.6/site-packages/distributed/client.py", line 592, in sync
return sync(self.loop, func, *args, **kwargs)
File "/anaconda3/lib/python3.6/site-packages/distributed/utils.py", line 254, in sync
six.reraise(*error[0])
File "/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/anaconda3/lib/python3.6/site-packages/distributed/utils.py", line 238, in f
result[0] = yield make_coro()
File "/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/anaconda3/lib/python3.6/site-packages/distributed/client.py", line 1315, in _gather
traceback)
File "/anaconda3/lib/python3.6/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/anaconda3/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 59, in loads
return pickle.loads(x)
ModuleNotFoundError: No module named 'diya_info'
如果我取消对@delayed和其他评论的评论，它将起作用。但是如何通过传递client作为参数来使其工作。
想法是对我要编写的所有库使用相同的客户端。
更新1：
我通过移除second problem装饰器并将函数放在@staticmethod中来修复fit closure。 但是@staticmethod有什么问题，这些装饰器是用于非自相关的东西的，对吧？
这里是diya_info.py：
...
def fit(self, X, y):
self.X = X
self.y = y
# function removed from @staticmethod
def has_null(df): return df.isnull().values.any()
# function removed from @staticmethod
def get_len(df): return len(df)
X_status = self.client.submit(has_null, self.X)
y_status = self.client.submit(has_null, self.y)
...
是否可以使用@staticmethod来做到这一点。我对解决此问题的方式不满意。对于Problem 1
答案 0 :(得分：1)
ModuleNotFoundError: No module named 'diya_info'
这意味着虽然您的客户端可以访问此模块，但您的工作人员却没有权限。解决此问题的一种简单方法是将脚本上传到您的工作人员。
client.upload_file('diya_info.py')
但总的来说，要确保您的工作人员和客户都具有相同的软件环境