假设我有一个包含贷款信息的pandas DataFrame，并且我想预测用户将不会退还钱的可能性（由我的数据框中的default列指示）。我想使用sklearn.model_selection.train_test_split将数据分为训练集和测试集。
但是，我要确保具有相同客户ID的贷款不会同时出现在测试和训练中。我应该怎么做？
下面是我的数据示例：
d = {'loan_date': ['20170101','20170701','20170301','20170415','20170515'],
'customerID': [111,111,222,333,444],
'loanID': ['aaa','fff','ccc','ddd','bbb'],
'loan_duration' : [6,3,12,5,12],
'gender':['F','F','M','F','M'],
'loan_amount': [20000,10000,30000,10000,40000],
'default':[0,1,0,0,1]}
df = pd.DataFrame(data=d)
CustomerID==111贷款记录应该出现在测试或火车中，但不能同时出现在两者中。
答案 0 :(得分：0)
我提出以下解决方案。对于具有相同customerID的客户，不会出现在培训和测试中；按活动划分的aslo客户-即将大约相同比例的具有相同数量贷款的用户接受培训和测试。
出于演示目的，我扩展了数据样本：
d = {'loan_date': ['20170101','20170701','20170301','20170415','20170515','20170905', '20170814', '20170819', '20170304'],
'customerID': [111,111,222,333,444,222,111,444,555],
'loanID': ['aaa','fff','ccc','ddd','bbb','eee', 'kkk', 'zzz', 'yyy'],
'loan_duration' : [6,3,12,5,12, 3, 17, 4, 6],
'gender':['F','F','M','F','M','M', 'F', 'M','F'],
'loan_amount': [20000,10000,30000,10000,40000,20000,30000,30000,40000],
'default':[0,1,0,0,1,0,1,1,0]}
df = pd.DataFrame(data=d)
代码：
from sklearn.model_selection import train_test_split
def group_customers_by_activity(df):
value_count = df.customerID.value_counts().reset_index()
df_by_customer = df.set_index('customerID')
df_s = [df_by_customer.loc[value_count[value_count.customerID == count]['index']] for count in value_count.customerID.unique()]
return df_s
-此功能按customerID活动（具有相同customerID的条目数）划分df。
此函数的示例输出：
group_customers_by_activity(df)
Out:
[ loan_date loanID loan_duration gender loan_amount default
customerID
111 20170101 aaa 6 F 20000 0
111 20170701 fff 3 F 10000 1
111 20170814 kkk 17 F 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
333 20170415 ddd 5 F 10000 0
555 20170304 yyy 6 F 40000 0]
-具有1，2，3贷款等的用户组。
此功能以用户到达火车或测试以下方式的方式来拆分组：
def split_group(df_group, train_size=0.8):
customers = df_group.index.unique()
train_customers, test_customers = train_test_split(customers, train_size=train_size)
train_df, test_df = df_group.loc[train_customers], df_group.loc[test_customers]
return train_df, test_df
split_group(df_s[2])
Out:
( loan_date loanID loan_duration gender loan_amount default
customerID
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0)
其余的将其应用于“客户活动”的所有组：
def get_sized_splits(df_s, train_size):
train_splits, test_splits = zip(*[split_group(df_group, train_size) for df_group in df_s])
return train_splits, test_splits
df_s = group_customers_by_activity(df)
train_splits, test_splits = get_sized_splits(df_s, 0.8)
train_splits, test_splits
Out:
((Empty DataFrame
Columns: [loan_date, loanID, loan_duration, gender, loan_amount, default]
Index: [],
loan_date loanID loan_duration gender loan_amount default
customerID
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
333 20170415 ddd 5 F 10000 0),
( loan_date loanID loan_duration gender loan_amount default
customerID
111 20170101 aaa 6 F 20000 0
111 20170701 fff 3 F 10000 1
111 20170814 kkk 17 F 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0,
loan_date loanID loan_duration gender loan_amount default
customerID
555 20170304 yyy 6 F 40000 0))
不要担心emty DataFrame，它将很快被串联。 split函数具有以下定义：
def split(df, train_size):
df_s = group_customers_by_activity(df)
train_splits, test_splits = get_sized_splits(df_s, train_size=train_size)
return pd.concat(train_splits), pd.concat(test_splits)
split(df, 0.8)
Out[106]:
( loan_date loanID loan_duration gender loan_amount default
customerID
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1
555 20170304 yyy 6 F 40000 0,
loan_date loanID loan_duration gender loan_amount default
customerID
111 20170101 aaa 6 F 20000 0
111 20170701 fff 3 F 10000 1
111 20170814 kkk 17 F 30000 1
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0
333 20170415 ddd 5 F 10000 0)
-因此，customerID放置在火车或测试数据中。我猜是因为输入数据的大小小，所以出现了这样的裂隙（火车>测试）。
如果您不需要按“ customerID活动”进行分组，则可以忽略它，而只需使用split_group即可达到目标。