Question

我正在使用Kaggle的泰坦尼克号教程熟悉scikit及其pandas集成。我已经清理了我的数据并希望做出一些预测。我可以调用管道拟合和转换 - 不幸的是我在尝试使用cross_val_score时遇到错误。

我正在使用sklearn-pandas cross_val_score

代码如下：

mapping = [
        ('Age', None),
        ('Embarked',LabelBinarizer()),
        ('Fare',None),
        ('Pclass',LabelBinarizer()),
        ('Sex',LabelBinarizer()),
        ('Group',LabelBinarizer()),
        ('familySize',None),
        ('familyType',LabelBinarizer()),
        ('Title',LabelBinarizer())
    ]


pipe = Pipeline([
    ('featurize', DataFrameMapper(mapping)), 
    ('logReg', LogisticRegression())
    ])

X = df_train[df_train.columns.drop('Survived')]
y = df_train['Survived']

#model = pipe.fit(X = X, y = y)
#prediction = model.predict(df_train)

score = cross_val_score(pipe, X = X, y = y, scoring = 'accuracy')

df_train是一个Pandas数据框，包含我的所有训练集，包括结果。两条评论说：

model = pipe.fit(X = X, y = y)
prediction = model.predict(df_train)

工作正常，预测会给我一个预测结果的数组。使用与cross_val_score相同的，我收到以下错误：

X has 20 features per sample; expecting 19

以下完整代码，可以使用Kaggle（https://www.kaggle.com/c/titanic/data）上的泰坦尼克号CSV文件运行

#%% Libraries import
import pandas as pd
import numpy as np
from sklearn_pandas import DataFrameMapper, cross_val_score 

from sklearn.preprocessing import LabelBinarizer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

#%% Read the data

path = 'E:/Kaggle/Titanic/Data/'
file_training = 'train.csv'
file_test = 'test.csv'

#Import the training and test dataset and concatenate them
df_training = pd.read_csv(path + file_training, header = 0, index_col = 'PassengerId')
df_test = pd.read_csv(path + file_test, header = 0, index_col = 'PassengerId')

# Work on the concatenated training and test data for feature engineering and clean-up
df = pd.concat([df_training, df_test], keys = ['train','test'])


#%% Initial data exploration and cleaning

df.describe(include = 'all')
pd.isnull(df).sum() > 0

#%% Preprocesing and Cleanup

#Create new columns with the name (to identify individuals part of a family)
df['LName'] = df['Name'].apply(lambda x:x.split(',')[0].strip())
df['FName'] = df['Name'].apply(lambda x:x.split(',')[1].split('.')[1].strip())

#Get the title
df['Title'] = df['Name'].apply(lambda x:x.split(',')[1].split('.')[0].strip())

titleDic = { 
        'Master' : 'kid',
        'Mlle' : 'unmarriedWoman',
        'Miss' : 'unmarriedWoman',
        'Ms' : 'unmarriedWoman',
        'Jonkheer' : 'noble',
        'Don' : 'noble',
        'Dona' : 'noble',
        'Sir' : 'noble',
        'Lady' : 'noble',
        'the Countess' : 'noble',
        'Capt' : 'ranked',
        'Major' : 'ranked',
        'Col' : 'ranked',
        'Mr' : 'standard',
        'Mme' : 'standard',
        'Mrs' : 'standard',
        'Dr' : 'academic',
        'Rev' : 'academic'
        }
df['Group'] = df['Title'].map(titleDic)

#%% Working with the family size

#Get the family size
df['familySize'] = df['Parch'] + df['SibSp'] + 1

#Add a family tag (single, couple, small, large)
df['familyType'] = pd.cut(df['familySize'], 
  [1,2,3,5,np.inf], 
  labels = ['single','couple','sFamily','bFamily'], 
  right = False)

#%% Filling empty values
#Fill empty values with the mean or mode for the column

#Fill the missing values with mean for age per title, class and gender. Store value in AgeFull variable
agePivot = pd.DataFrame(df.groupby(['Group', 'Sex'])['Age'].median())
agePivot.columns = ['AgeFull']
df = pd.merge(df, agePivot, left_on = ['Group', 'Sex'], right_index = True)
df.loc[df['Age'].isnull(),['Age']] = df['AgeFull']

#Embark location missing values
embarkPivot = pd.DataFrame(df.groupby(['Group'])['Embarked'].agg(lambda x:x.value_counts().index[0]))
embarkPivot.columns = ['embarkFull']
df = pd.merge(df, embarkPivot, left_on = ['Group'], right_index = True)
df.loc[df['Embarked'].isnull(),['Embarked']] = df['embarkFull']

#Fill the missing fare value
df.loc[df['Fare'].isnull(), 'Fare'] = df['Fare'].mean()

#%% Final clean-up (drop temporary columns)

df = df.drop(['AgeFull', 'embarkFull'], 1)

#%% Preparation for training

df_train = df.loc['train']
df_test = df.loc['test']

#Creation of dummy variables

mapping = [
            ('Age', None),
            ('Embarked',LabelBinarizer()),
            ('Fare',None),
            ('Pclass',LabelBinarizer()),
            ('Sex',LabelBinarizer()),
            ('Group',LabelBinarizer()),
            ('familySize',None),
            ('familyType',LabelBinarizer()),
            ('Title',LabelBinarizer())
        ]

pipe = Pipeline(steps = [
        ('featurize', DataFrameMapper(mapping)), 
        ('logReg', LogisticRegression())
        ])

#Uncommenting the line below fixes the code - why?
#df_train = df_train.sort_index()

X = df_train[df_train.columns.drop(['Survived'])]
y = df_train.Survived

score = cross_val_score(pipe, X = df_train, y = df_train.Survived, scoring = 'accuracy')

Answer 1

这非常有趣。我已经解决了这个问题，只需在将数据传递给管道中的cross_val_score之前使用DataFrame进行排序。

df_train = df_train.sort_index()

有人能解释一下为什么这会对Scikit的工作方式产生影响吗？

Scikit-pandas，cross val得分特征

1 个答案: