Question

我正在对我拥有的数据进行文本分类。根据一些观察，我需要确定目标变量。我从一揽子单词和tf / idf方法开始。

我使分类器具有“一个”功能，但是当我尝试结合更多“功能”（例如7）来预测标签时，Count Vectorizer会为fit_transform引发错误。以下是代码

from sklearn import preprocessing
from sklearn.feature_extraction.text import CountVectorizer    
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import LogisticRegression

bow = CountVectorizer()
# working fine for one feature
#observation = df_all_null_removed['Observation'].values
# selecting feature set of 7 variables
observation = df_all_null_removed[features].values

train_obs, test_obs,y_train, y_test =train_test_split(observation,
                               df_all_null_removed['HazardType'],                                                       
                               test_size=0.12,
                               random_state=42)
bow_matrix = bow.fit_transform(observation) # throws error - screen shot attached.

我认为这是因为“观测”是一个二维[8150,7]形状的numpy数组，我们需要将其转换为8150行的1列数组。

AttributeError                            Traceback (most recent call last)
<ipython-input-140-d75b27bd1080> in <module>()
----> 1 bow_matrix = bow.fit_transform(observation)
      2 print("The vocabulary of the bow",len(bow.vocabulary_))

~/anaconda3/lib/python3.6/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
    867 
    868         vocabulary, X = self._count_vocab(raw_documents,
--> 869                                           self.fixed_vocabulary_)
    870 
    871         if self.binary:

~/anaconda3/lib/python3.6/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
    790         for doc in raw_documents:
    791             feature_counter = {}
--> 792             for feature in analyze(doc):
    793                 try:
    794                     feature_idx = vocabulary[feature]

~/anaconda3/lib/python3.6/site-packages/sklearn/feature_extraction/text.py in <lambda>(doc)
    264 
    265             return lambda doc: self._word_ngrams(
--> 266                 tokenize(preprocess(self.decode(doc))), stop_words)
    267 
    268         else:

~/anaconda3/lib/python3.6/site-packages/sklearn/feature_extraction/text.py in <lambda>(x)
    230 
    231         if self.lowercase:
--> 232             return lambda x: strip_accents(x.lower())
    233         else:
    234             return strip_accents

AttributeError: 'numpy.ndarray' object has no attribute 'lower'

Answer 1

您可以使用ColumnTransformer为数据提供多种不同的预处理途径。

如何在使用CountVectorizer进行训练数据集时解决“ numpy.ndarray”错误？

1 个答案: