Question

目标是使用三个输入来运行多标签分类器。每个输入都是较大文档的摘录。该管道有一个初步步骤，可以使用tfidf对每个摘录进行矢量化处理

x是一个字符串列表，每个字符串都摘录。

下面的代码有效，但似乎忽略了列表的第二和第三元素。

def grid_search(train_x, train_y):

    from sklearn.model_selection import GridSearchCV
    from sklearn.pipeline import Pipeline
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.multiclass import OneVsRestClassifier
    from sklearn.naive_bayes import MultinomialNB


    parms={ 'tfidf__max_df': (0.25, 0.5, 0.75),
            'tfidf__ngram_range': [(1, 1), (1, 2), (1, 3)],
            'clf__estimator__alpha': (1e-2, 1e-3)
            } 
    tfidf1 = ('tfidf', TfidfVectorizer(stop_words=stop_words))
    vctrz= tfidf1 
    clsfy = ('clf', OneVsRestClassifier(MultinomialNB( fit_prior=True, class_prior=None)))
    pipeline = Pipeline([ vctrz, clsfy ])


    gs1 = GridSearchCV(pipeline, parms, cv=2, n_jobs=1, verbose=0)
    gs1.fit(train_x, train_y)
    return gs1.best_estimator_


classifier = grid_search(train_x, y_train)

我尝试没有成功

    vctrz = [tfidf1,tfidf1,tfidf1]

我也尝试过FeatureUnion

TFALL =  [('tf1', TFIDFX1()) , ('tf2', TFIDFX2()) , ('tf3', TFIDFX3()) ]
#maybe the () are extraneous but without them I get a self less error

clsfy = ('clf', OneVsRestClassifier(MultinomialNB( fit_prior=True, class_prior=None)))

ppl = Pipeline([     ('feats', FeatureUnion(TFALL) ),   clsfy    ])


gs1 = GridSearchCV(ppl, parms, cv=2, n_jobs=1, verbose=5)

其中TFIDFX1的构造如下

class TFIDFX1(BaseEstimator, TransformerMixin):


    def __init__(self):
        pass

    def vectorize(self, doc):
        return vect.fit(doc)

    def transform(self, mylist, y=None):
        return self.vectorize(mylist[0]) #would 

    def fit(self, df, y=None):
        return self

为简洁起见，我省略了TFIDFX2和TFIDFX3类，它们分别看起来与mylist [1]和mylist [2]相似，但在其他方面相同

此操作失败，并带有以下回溯：

TypeError: float() argument must be a string or a number, not 'TfidfVectorizer'

我将非常感谢SO社区的帮助

Answer 1

即使三个输入是同质的，tfidf步骤也不会自动跨阵列

相反，您必须使用Featureunion步骤并将三个输入组合为三个独立的tfidf子步骤，如this example

感谢@Vivek Kumar

sklearn特征联合

1 个答案: