我正在研究Kaggle主持的房价问题。在构建模型时，我发现在测试集上重用我一直用于训练数据集的某些代码也是有意义的，因此我将执行互操作的代码合并到一个函数定义中。在此函数中，我要处理缺失的值，并使用其返回值执行一次热编码，并在随机森林回归中使用它。但是，它引发以下错误：
Traceback (most recent call last):
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 56, in <module>
sel.fit(x_train, y_train)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\feature_selection\from_model.py", line 196, in fit
self.estimator_.fit(X, y, **fit_params)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\forest.py", line 249, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
在使用相同代码而不将其组织成一个函数时，我没有遇到这个问题。 def feature_selection_and_engineering(df)是有问题的功能。以下是我的完整代码：
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
def feature_selection_and_engineering(df):
# Creating a series of how many NaN's are in each column
nan_counts = df.isna().sum()
# Creating a template list
nan_columns = []
# Iterating over the series and if the value is more than 0 (i.e there are some NaN's present)
for i in range(0, len(nan_counts)):
if nan_counts[i] > 0:
nan_columns.append(df.columns[i])
# Iterating through all the columns which are known to have NaN's
for i in nan_columns:
if df[nan_columns][i].dtypes == 'float64':
df[i] = df[i].fillna(df[i].mean())
elif df[nan_columns][i].dtypes == 'object':
df[i] = df[i].fillna('XX')
# Creating a template list
categorical_columns = []
# Iterating across all the columns,
# checking if they're of the object datatype and if they are, appending them to the categorical list
for i in range(0, len(df.dtypes)):
if df.dtypes[i] == 'object':
categorical_columns.append(df.columns[i])
return categorical_columns
# take one-hot encoding
OHE_sdf = pd.get_dummies(feature_selection_and_engineering(train))
# drop the old categorical column from original df
train.drop(columns = feature_selection_and_engineering(train), axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
# Dividing the training dataset into train/test sets with the test size being 20% of the overall dataset.
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.2, random_state = 42)
randomForestRegressor = RandomForestRegressor(n_estimators=1000)
# Invoking the Random Forest Classifier with a 1.25x the mean threshold to select correlating features
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
selected = sel.get_support()
# linearRegression.fit(x_train, y_train)
randomForestRegressor.fit(x_train, y_train)
# Assigning the accuracy of the model to the variable "accuracy"
accuracy = randomForestRegressor.score(x_train, y_train)
# Predicting for the data in the test set
predictions = randomForestRegressor.predict(feature_selection_and_engineering(test))
# Writing the predictions to a new CSV file
submission = pd.DataFrame({'Id': test['PassengerId'], 'SalePrice': predictions})
filename = 'Boston-Submission.csv'
submission.to_csv(filename, index=False)
print(accuracy*100, "%")
新错误：
Traceback (most recent call last):
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 76, in <module>
x_train, encoder = feature_selection_and_engineering(x_train)
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 57, in feature_selection_and_engineering
encoder = train_one_hot_encoder(df, categorical_columns)
File "/home/onur/Documents/Boston-Kaggle/Model.py", line 30, in train_one_hot_encoder
return enc.fit(categorical_df)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 493, in fit
self._fit(X, handle_unknown=self.handle_unknown)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 80, in _fit
X_list, n_samples, n_features = self._check_X(X)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 67, in _check_X
force_all_finite=needs_validation)
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 60, in _assert_all_finite
raise ValueError("Input contains NaN")
ValueError: Input contains NaN
答案 0 :(得分：1)
重用代码是一个好主意，但是请注意，当您将代码放入函数中时，变量的范围将如何变化。
由于在数组中输入NaN到随机目录林中的值，导致出现错误。在feature_engineering_and_selection()函数中，您要删除NaN的值，但是函数不会返回df，因此在模型中使用了未经修改的原始df。 / p>
我建议将feature_engineering_and_selection()函数拆分为不同的组件。在这里，我做了一个仅删除NaN的函数。
# Iterates through the columns and fixes any NaNs
def remove_nan(df):
replace_dict = {}
for col in df.columns:
# If there are any NaN values in this column
if pd.isna(df[col]).any():
# Replace NaN in object columns with 'N/A'
if df[col].dtypes == 'object':
replace_dict[col] = 'N/A'
# Replace NaN in float columns with 0
elif df[col].dtypes == 'float64':
replace_dict[col] = 0
df = df.fillna(replace_dict)
return df
我建议用0而不是平均值填充NaN数值。对于此数据，存在3个带有nan值的数字列：LotFrontage（与房屋相连的街道英尺），MasVnrArea（砖石装饰面积）和GarageYrBlt（建成车库的年份）。如果没有车库，则没有建造车库的年份，因此将年份设为0而不是平均年份等是有意义的。
使用已设置的一个热编码器还需要完成一些工作。创建单热编码可能很棘手，因为训练数据和测试数据必须具有相同的列。如果您具有以下培训和测试数据
火车
| House Type |
| ---------- |
| Mansion |
| Ranch |
测试
| House Type |
| ---------- |
| Mansion |
| Duplex |
然后，如果使用pd.get_dummies()，则训练列将为[house_type_mansion, house_type_ranch]，测试列将为[house_type_mansion, house_type_duplex]，这将不起作用。但是，使用sklearn，您可以在火车数据中安装一个热编码器。转换测试数据集时，它将创建与火车数据集相同的列。 handle_unknown参数将告诉编码器如何处理duplex或ignore测试集中的error。
# Fits an sklearn one hot encoder
def train_one_hot_encoder(df, categorical_columns):
# take one-hot encoding of categorical columns
categorical_df = df[categorical_columns]
enc = OneHotEncoder(sparse=False, handle_unknown='ignore')
return enc.fit(categorical_df)
再次结合分类和非分类数据，我建议创建一个单独的函数
# One hot encodes the given dataframe
def one_hot_encode(df, categorical_columns, encoder):
# Get dataframe with only categorical columns
categorical_df = df[categorical_columns]
# Get one hot encoding
ohe_df = pd.DataFrame(encoder.transform(categorical_df), columns=encoder.get_feature_names())
# Get float columns
float_df = df.drop(categorical_columns, axis=1)
# Return the combined array
return pd.concat([float_df, ohe_df], axis=1)
最后，您的feature_engineering_and_selection()函数可以调用所有这些函数。
def feature_selection_and_engineering(df, encoder=None):
df = remove_nan(df)
categorical_columns = get_categorical_columns(df)
# If there is no encoder, train one
if encoder == None:
encoder = train_one_hot_encoder(df, categorical_columns)
# Encode Data
df = one_hot_encode(df, categorical_columns, encoder)
# Return the encoded data AND encoder
return df, encoder
为了使代码运行，我必须修复一些问题，这里https://gist.github.com/kylelrichards11/6be90d92a7dd6a5cc9a5290dae3ff94e
包含了整个修改后的脚本。