Question

给出一个DataFrame，其中只有一列Text：

      Text
0     chest  pain  nstemi  this  84-year  old  man  present  on  26/5  with  
      chest  pain  associate  with  profuse  sweating  and  nausea

我想创建两个新列，其中包含为前一个DataFrame生成的字母组合和二字组。

这是我用来生成ngram的方法：

    def generate_ngrams(self, s, n):
        # Convert to lowercases
        s = s.lower()

        # Replace all none alphanumeric characters with spaces
        s = re.sub(r'[^a-zA-Z0-9\s]', ' ', s)

        # Break sentence in the token, remove empty tokens
        tokens = [token for token in s.split(" ") if token != ""]

        # Use the zip function to help us generate n-grams
        # Concatentate the tokens into ngrams and return
        ngrams = zip(*[tokens[i:] for i in range(n)])
        return [" ".join(ngram) for ngram in ngrams]

这就是我试图填充我的DataFrame的方式：

    for index, row in featuresDF.iterrows():
        featuresDF.at[index, '1-gram'] = generate_ngrams(infoDF.at[index, 'Text'], 1)
        featuresDF.at[index, '2-gram'] = generate_ngrams(infoDF.at[index, 'Text'], 2)

运行它时，出现以下错误：ValueError: setting an array element with a sequence.

这是回溯：

Traceback (most recent call last):

  File "<ipython-input-64-e014e2e1c7e2>", line 3, in <module>
    featuresDF.at[index, '1-gram'] = featureExtraction.generate_ngrams(infoDF.at[index, 'Text'], 1)

  File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\indexing.py", line 2287, in __setitem__
    self.obj._set_value(*key, takeable=self._takeable)

  File "C:\Users\as\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py", line 2815, in _set_value
    engine.set_value(series._values, index, value)

  File "pandas/_libs/index.pyx", line 95, in pandas._libs.index.IndexEngine.set_value

  File "pandas/_libs/index.pyx", line 106, in pandas._libs.index.IndexEngine.set_value

我知道当我向DataFrame分配unigram和bigram时，这是一个问题，对吗？但我不确定如何解决。谢谢！

Answer 1

generate_ngrams（）应该返回一个字符串，但返回的列表如下：

['chest', 'pain', .....]

在返回列表之前，您无法将其转换为逗号分隔的字符串，例如：

chest,pain, .....

通过添加以下行：

ngramList = [" ".join(ngram) for ngram in ngrams]        
return ','.join(ngramList)

此外，您可以使用CountVectorizer查找N-gram：

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(ngram_range=(2,2)) # 2,2 means 2-gram, 1,1 is unigram
corpus = ['the boy is gone !']
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names()) # this will print the list containing gram values

Answer 2

您正在使用return [" ".join(ngram) for ngram in ngrams]

返回列表

仅返回字符串本身，而不是返回列表：

return " ".join(ngram) for ngram in ngrams

如果您确实希望使用列表设置元素，请遵循此ValueError: setting an array element with a sequence. for Pandas

尝试从DataFrame列创建n-gram时出错

2 个答案: