Question

我试图用一个玩具示例来说明我的困惑。我意识到这是虹膜数据集的一个愚蠢的例子。

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin


class ItemSelector(BaseEstimator, TransformerMixin):
    """For data grouped by feature, select subset of data at a provided key.

    The data is expected to be stored in a 2D data structure, where the first
    index is over features and the second is over samples.  i.e.

    >> len(data[key]) == n_samples

    Please note that this is the opposite convention to scikit-learn feature
    matrixes (where the first index corresponds to sample).

    ItemSelector only requires that the collection implement getitem
    (data[key]).  Examples include: a dict of lists, 2D numpy array, Pandas
    DataFrame, numpy record array, etc.

    >> data = {'a': [1, 5, 2, 5, 2, 8],
               'b': [9, 4, 1, 4, 1, 3]}
    >> ds = ItemSelector(key='a')
    >> data['a'] == ds.transform(data)

    ItemSelector is not designed to handle data grouped by sample.  (e.g. a
    list of dicts).  If your data is structured this way, consider a
    transformer along the lines of `sklearn.feature_extraction.DictVectorizer`.

    Parameters
    ----------
    key : hashable, required
        The key corresponding to the desired value in a mappable.
    """
    def __init__(self, key):
        self.key = key

    def fit(self, x, y=None):
        return self

    def transform(self, data_dict):
        return data_dict[self.key]

# 150 examples, 4 features, labels in {0, 1, 2}
iris = load_iris()
y = iris.target
dfX = pd.DataFrame(iris.data, columns=iris.feature_names)

# feature union transformer list 
transformer_list = [
    ('sepal length (cm)', Pipeline([
        ('selector', ItemSelector(key='sepal length (cm)')),
    ])),

    ('sepal width (cm)', Pipeline([
        ('selector', ItemSelector(key='sepal width (cm)')),
    ])),

    ('petal length (cm)', Pipeline([
        ('selector', ItemSelector(key='petal length (cm)')),
    ])),

    ('petal width (cm)', Pipeline([
        ('selector', ItemSelector(key='petal width (cm)')),
    ])),
]

# create pipeline
pipeline = Pipeline([
    ("union", FeatureUnion(transformer_list=transformer_list)),
    ("svm", SVC(kernel="linear")),
])

# train model
param_grid = dict({})
search = GridSearchCV(estimator=pipeline, param_grid=param_grid, n_jobs=1)
search.fit(dfX, y)
print(search.best_estimator_)

出错：

/Users/me/.virtualenvs/myenv/lib/python2.7/site-packages/sklearn/utils/validation.pyc in check_consistent_length(*arrays)
    179     if len(uniques) > 1:
    180         raise ValueError("Found input variables with inconsistent numbers of"
--> 181                          " samples: %r" % [int(l) for l in lengths])
    182
    183

ValueError: Found input variables with inconsistent numbers of samples: [1, 99]

我的想法是FeatureUnions是并行的，Pipelines是串行的。

我怎么想这个错？混合到两个以获得某些要素类型的丰富流量的正确方法是什么，并且能够分段添加变换器，但仍然将所有变量与FeatureUnion结合在一起用于最终预测变量？

Answer 1

是的，你的想法是正确的：

我的想法是FeatureUnions是并行的，Pipelines是串行的。

但问题是你的ItemSelector正在返回一个形状为(150,)的numpy数组。在FeatureUnion的最后一步中，完成了来自不同变换器的功能的连接，使用numpy.hstack()水平堆叠数组。请注意，输出中不存在第二个维度。因此，当为所有变换器完成并集时，它会生成一个形状为(600,)的数组。这就是您在fit方法中收到错误的原因。

返回数组的正确形状应为(150,4)。您必须再向管道添加一个变换器，以便以正确的形状返回数据（或使用numpy.asmatrix()和transpose()手动更改返回数据的形状）。

请查看FeatureUnion的文档示例。在此示例中，ItemSelector的输出传递给其他变换器，如TfidfVectorizer()或DictVectorizer()，它们返回2d数组中的数据，然后在FeatureUnion中正确合并。希望它有所帮助。

关于Sklearn管道和混乱的困惑功能联盟在一起

1 个答案: