Question

我有一个句子列表。我想随机分为80％和20％，如下所示：

['Hi.',
 'Hi.',
 'Run!',
 'Wow!',
 'Wow!',
 'Fire!',
 'Help!',
 'Help!',
 'Stop!',
 'Wait!',
 'Go on.',
 'Hello!',
 'I ran.',
 'I see.',
 'I see.',
 'I try.',
 'I won!',...]

我在想戴口罩

import random
mask = [0] * 4000 + [1] * 16000
random.shuffle(mask)

但是它不像数据帧。我尝试了

percent=80
bol_mask =[random.randrange(100) < percent for i in range(100)]

不能将布尔值应用于句子

还必须保留分隔掩码，以后将其应用于另一个德语列表，即相应的翻译。

看起来像这样

array([[ 553,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3430, 1114,    6,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [1115,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3431,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3432,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [2459,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3433,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [1533, 3434,    6,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [2460,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [ 394,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0]],
      dtype=int32)

我的问题是如何将蒙版应用于句子列表？并保持相同的分割并应用于相应的ndarray？

Answer 1

如果可以选择使用scikit-learn，则可以按照以下方式使用train_test_split方法：

>>> from sklearn.model_selection import train_test_split
>>> print(x)
>>> x
['Hi.', 'Hi.', 'Run!', 'Wow!', 'Wow!', 'Fire!', 'Help!', 'Help!', 'Stop!', 'Wait!']

>>> len(x)
10
>>>  x1
array([[ 553,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3430, 1114,    6,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [1115,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3431,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3432,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [2459,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [3433,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [1533, 3434,    6,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [2460,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0],
       [ 394,    6,    0,    0,    0,    0,    0,    0,    0,    0,    0,
           0,    0,    0,    0,    0,    0,    0,    0,    0]])
>>> x1.shape
(10, 20)

#assuming x, x1 have same length train test split should work fine.
>>> train, test, train_german, test_german = train_test_split(x,x1, test_size=0.2, shuffle=True)
>>> len(train)
8
>>> len(test)
2
>>> len(train_german)
8
>>> len(test)
2

Answer 2

实际上我已经解决了自己的问题。

bol_mask =[random.randrange(100) < 80 for i in range(20000)]
inv_mask = np.invert(bol_mask)

Eng_train =np.array(Eng)[bol_mask]
Eng_test =np.array(Eng)[inv_mask]
German_train = padded[bol_mask]
German_test = padded[inv_mask]

感谢Grayrigel，感谢您在帮助方面的努力

火车测试拆分了一个句子列表

2 个答案: