Question

我正在尝试在sklearn中设置自定义计分器（使用make_scorer）以在交叉验证期间使用。具体来说，我想为一个多类别分类示例计算Top2准确性。

从技术上讲，我的问题是，我需要评估概率（使用 needs_proba = True ），并需要列出类别，以便理解概率矩阵。

我在下面整理了一个例子。虽然可以通过在 make_scorer 调用中提供类来为非cv示例设置自定义评分功能，但是我无法为cv-case正确设置该类，因为动态确定，因此我只需要在评估过程中阅读它们即可。

我知道有很多类似的问题，但是我没有看到针对我的特定用例的可行解决方案，因此如果有人可以帮助我（如果在某个地方解决了这个问题，请原谅我的无知），我将非常高兴。

非常感谢！大卫

PS：如果我没记错的话，对于所有涉及概率的make_scorer用例，实际上类标签都至关重要，因此我认为这是一个普遍问题。

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_validate

data = load_iris()
X = data.data
y = data.target

# DIRECT USE OF CUSTOM SCORER ##################################################################################
# Simple test train split
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Define the model and fit it
model = LogisticRegression()
model.fit(X = X_train, y = y_train)

# Function that returns either the prediction with the highest likelihood or the correct prediction, 
# if it is within Top n by probability 
def top_n_consolidation(y_label, y_prob, class_names, n=2):
    top_recs = class_names[[i[0] for i in sorted(enumerate(y_prob), key=lambda x:x[1], reverse=True)][0:n]]
    if any([i == y_label for i in top_recs]):
        return y_label
    else:
        return top_recs[0]

# Calculate accuracy based on Top n predictions
# --> NOTE: THIS FUNCTION RELIES ON class_names IN ORDER TO MAKE USE OF THE PROBABILITIES
def accuracy_top_n_function(y_labels, y_probs, class_names, n=2):
    cons_preds = [top_n_consolidation(y_labels[i], y_probs[i,:], class_names, n) for i in range(y_probs.shape[0])]
    return accuracy_score(y_true=y_labels, y_pred=cons_preds)

# Make a custom scorer for Top 2 classifications
accuracy_2 = make_scorer(accuracy_top_n_function, class_names = model.classes_, n=2, needs_proba = True)
# --> NOTE: THIS WORKS, BECAUSE model.fit WAS ALREADY EXECUTED

# Calculate Top 2 accuracies
accuracy_2(clf=model, X=X_test, y=y_test)

# USE OF CUSTOM SCORER FOR CROSS-VALIDATION ####################################################################

# Define a new model to ensure that we distinguish from the case above
model_cv = LogisticRegression()

# Define custom scorer for the cv case
accuracy_2_cv = make_scorer(accuracy_top_n_function, class_names = model_cv.classes_, n=2, needs_proba = True)
# NOTE: THIS IS NOT WORKING AS model_cv.classes_ IS NOT YET KNOWN!

# Define custom scores to use
custom_scoring = {'acc'       : 'accuracy',
                  'acc2'      : accuracy_2_cv}

cross_validate(model_cv, X, y, cv=3, scoring = custom_scoring, return_train_score=True)

Answer 1

您可以使用custom scoring method described here in user guide，其签名为：

func(estimator, X, y)

这里estimator是一个拟合估计器，具有来自交叉验证拆分的火车数据，因此estimator.classes_可以工作。

def accuracy_2_cv(estimator, X, y_labels):
    n=2
    y_probs = estimator.predict_proba(X)
    class_names = estimator.classes_
    cons_preds = [top_n_consolidation(y_labels[i], y_probs[i,:], class_names, n) for i in range(y_probs.shape[0])]
    return accuracy_score(y_true=y_labels, y_pred=cons_preds)

现在将其直接传递到custom_scoring而不用包装在make_scorer中：

custom_scoring = {'acc'       : 'accuracy',
                  'acc2'      : accuracy_2_cv}

Sklearn：将类别名称传递给make_scorer

1 个答案: