Question

让我们假设我在DataFrame的{{1}}中具有以下内容：

pandas

我想将每个id的文本分成3个单词的标记，因此我最终想要拥有以下内容：

id  text
1   I am the first document and I am very happy.
2   Here is the second document and it likes playing tennis.
3   This is the third document and it looks very good today.

请记住，我的数据框可能还具有其他列，除了这两列外，其他列应该以与上述id text 1 I am the 1 first document and 1 I am very 1 happy 2 Here is the 2 second document and 2 it likes playing 2 tennis 3 This is the 3 third document and 3 it looks very 3 good today相同的方式简单地复制到新数据框中。

最有效的方法是什么？

我认为我的问题的解决方案与此处给出的解决方案非常接近：Tokenise text and create more rows for each row in dataframe。

这可能也有帮助：Python: Split String every n word in smaller Strings。

Answer 1

您可以使用类似的内容：

def divide_chunks(l, n): 
    # looping till length l 
    for i in range(0, len(l), n):  
        yield l[i:i + n]

然后使用unnesting：

df['text_new']=df.text.apply(lambda x: list(divide_chunks(x.split(),3)))
df_new=unnesting(df,['text_new']).drop('text',1)
df_new.text_new=df_new.text_new.apply(' '.join)
print(df_new)

              text_new  id
0             I am the   1
0   first document and   1
0            I am very   1
0               happy.   1
1          Here is the   2
1  second document and   2
1     it likes playing   2
1              tennis.   2
2          This is the   3
2   third document and   3
2        it looks very   3
2          good today.   3

编辑：

m=(pd.DataFrame(df.text.apply(lambda x: list(divide_chunks(x.split(),3))).values.tolist())
.unstack().sort_index(level=1).apply(' '.join).reset_index(level=1))
m.columns=df.columns
print(m)

   id                 text
0   0             I am the
1   0   first document and
2   0            I am very
3   0               happy.
0   1          Here is the
1   1  second document and
2   1     it likes playing
3   1              tennis.
0   2          This is the
1   2   third document and
2   2        it looks very
3   2          good today.

Answer 2

一个自包含的解决方案，可能会慢一些：

# Split every n words
n = 3

# incase id is not index yet
df.set_index('id', inplace=True)

new_df = df.text.str.split(' ', expand=True).stack().reset_index()

new_df = (new_df.groupby(['id', new_df.level_1//n])[0]
                .apply(lambda x: ' '.join(x))
                .reset_index(level=1, drop=True)
         )

new_df是一个系列：

id
1               I am the
1     first document and
1              I am very
1                 happy.
2            Here is the
2    second document and
2       it likes playing
2                tennis.
3            This is the
3     third document and
3          it looks very
3            good today.
Name: 0, dtype: object

在单元格中分割文本并为令牌创建其他行

2 个答案: