Question

尝试遵循本文对不平衡分类进行过采样。我的班级比例大约是8：1。

https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets/notebook

我对管道+编码结构感到困惑。

训练/测试拆分后，您是否应该对样本进行过度采样？
- 如果是这样，您如何处理目标标签已从X删除的事实？我尝试保留它，然后执行过采样，然后在X_train / X_test上放置标签，并替换管道中的新训练集但是我收到错误消息“ 发现样本数量不一致的输入变量”，因为形状不一致，因为新的过采样df以50/50的标签分布加倍。

我理解这个问题，但是当想要执行过度采样以减少类不平衡时，如何解决这个问题呢？


    X = df
    #X = df.drop("label", axis=1)
    y = df["label"]

    X_train,\
    X_test,\
    y_train,\
    y_test = train_test_split(X,\
                              y,\
                              test_size=0.2,\
                              random_state=11,\
                              shuffle=True,\
                              stratify=target)

    target_count = df.label.value_counts()
    print('Class 1:', target_count[0])
    print('Class 0:', target_count[1])
    print('Proportion:', round(target_count[0] / target_count[1], 2), ': 1')

    target_count.plot(kind='bar', title='Count (target)');

    # Class count
    count_class_index_0, count_class_index_1 = X_train.label.value_counts()

    # Divide by class
    count_class_index_0 = X_train[X_train['label'] == '1']
    count_class_index_1 = X_train[X_train['label'] == '0']

    df_class_1_over = df_class_1.sample(count_class_index_0, replace=True)
    df_test_over = pd.concat([count_class_index_0, df_class_1_over], axis=0)

    print('Random over-sampling:')
    print(df_test_over.label.value_counts())

    Random over-sampling:
    1    12682
    0      12682

    df_test_over.label.value_counts().plot(kind='bar', title='Count (target)')

    # drop label for new X_train and X_test
    X_train_OS = df_test_over.drop("label", axis=1)
    X_test = X_test.drop("label", axis=1)

    print(X_train_OS.shape)
    print(X_test.shape)

    print(y_train.shape)
    print(y_test.shape)

    (25364, 9)
    (3552, 9)
    (14207,)
    (3552,)

    cat_transformer = Pipeline(steps=[
        ('cat_imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('cat_ohe', OneHotEncoder(handle_unknown='ignore'))])

    num_transformer = Pipeline(steps=[
        ('num_imputer', SimpleImputer(strategy='constant', fill_value=0)),
        ('num_scaler', StandardScaler())])

    text_transformer_0 = Pipeline(steps=[
        ('text_bow', CountVectorizer(lowercase=True,\
                                     token_pattern=SPLIT_PATTERN,\
                                     stop_words=stopwords))])
    # SelectKBest()
    # TruncatedSVD()

    text_transformer_1 = Pipeline(steps=[
        ('text_bow', CountVectorizer(lowercase=True,\
                                     token_pattern=SPLIT_PATTERN,\
                                     stop_words=stopwords))])
    # SelectKBest()
    # TruncatedSVD()

    FE = ColumnTransformer(
        transformers=[
            ('cat', cat_transformer, CAT_FEATURES),
            ('num', num_transformer, NUM_FEATURES),
            ('text0', text_transformer_0, TEXT_FEATURES[0]),
            ('text1', text_transformer_1, TEXT_FEATURES[1])])

    pipe = Pipeline(steps=[('feature_engineer', FE),
                         ("scales", MaxAbsScaler()),
                         ('rand_forest', RandomForestClassifier(n_jobs=-1, class_weight='balanced'))])

    random_grid = {"rand_forest__max_depth": [3, 10, 100, None],\
                  "rand_forest__n_estimators": sp_randint(10, 100),\
                  "rand_forest__max_features": ["auto", "sqrt", "log2", None],\
                  "rand_forest__bootstrap": [True, False],\
                  "rand_forest__criterion": ["gini", "entropy"]}

    strat_shuffle_fold = StratifiedKFold(n_splits=5,\
      random_state=123,\
      shuffle=True)

    cv_train = RandomizedSearchCV(pipe, param_distributions=random_grid, cv=strat_shuffle_fold)
    cv_train.fit(X_train_OS, y_train)

    from sklearn.metrics import classification_report, confusion_matrix
    preds = cv_train.predict(X_test)
    print(confusion_matrix(y_test, preds))
    print(classification_report(y_test, preds))

Answer 1

SMOTE可以很容易地（并且可以说是更优雅地）解决您遇到的问题。它易于使用，并允许您保留X_train, X_test, y_train, y_test中的train_test_split语法，因为它将同时在X和y上执行过采样。

from imblearn.over_sampling import SMOTE

X_train, X_test, y_train, y_test = train_test_split(X,y)
sm = SMOTE(random_state=42)
X_resampled, y_resampled = sm.fit_resample(X_train, y_train)

Answer 2

所以我相信我已经解决了自己的问题...问题是我如何拆分数据...我通常总是遵循标准的X_train，X_test，y_train，y_test train_test_split，但是这导致了行计数不匹配X_train和y_train进行过采样时，所以我改为这样做，一切似乎都正常了。如果有人有任何建议，请告诉我！谢谢！

features = df_
target = df_l["label"]

train_set, test_set = train_test_split(features, test_size=0.2,\
                          random_state=11,\
                          shuffle=True)

print(train_set.shape)
print(test_set.shape)

(11561, 10)
(2891, 10)

count_class_1, count_class_0 = train_set.label.value_counts()

# Divide by class
df_class_1 = train_set[train_set['label'] == 1]
df_class_0 = train_set[train_set['label'] == 0]

df_class_0_over = df_class_0.sample(count_class_1, replace=True)
df_train_OS = pd.concat([df_class_1, df_class_0_over], axis=0)

print('Random over-sampling:')
print(df_train_OS.label.value_counts())

1      10146
0    10146

df_train_OS.label.value_counts().plot(kind='bar', title='Count (target)');

X_train_OS = df_train_OS.drop("label", axis=1)
y_train_OS = df_train_OS["label"]
X_test = test_set.drop("label", axis=1)
y_test = test_set["label"]

print(X_train_OS.shape)
print(y_train_OS.shape)
print(X_test.shape)
print(y_test.shape)

(20295, 9)
(20295,)
(2891, 9)
(2891,)

过度采样的班级不平衡训练/测试拆分“找到的输入变量样本数量不一致”解决方案？

2 个答案: