如何在python中混淆数据,保持一些n行完好无损

时间:2017-11-16 07:15:31

标签: python-3.x pandas numpy

我想以这样的方式对我的数据进行洗牌,即每4行保持不变。例如,我有16行,然后前4行可以到最后,然后第二行可以到第三行,依此类推。我想在python

中做些什么

3 个答案:

答案 0 :(得分:3)

将第一个轴重新分成两个,后面的长度与组长度4相同,给我们一个3D数组然后使用np.random.shuffle,它沿着第一个轴移动轴。重新整形的版本是原始数组的视图,将结果直接分配给它。作为in-situ,这应该非常有效(无论是记忆还是性能)。

因此,实施将如此简单 -

def array_shuffle(a, n=4):
    a3D = a.reshape(a.shape[0]//n,n,-1) # a is input array
    np.random.shuffle(a3D)

它的另一个变体是生成覆盖3D数组长度的随机排列,然后用这些排列索引并最终重新转换回2D。这样就可以复制,但似乎更多如上一种方法所示,性能高于in-situ

实施将是 -

def array_permuted_indexing(a, n=4):
    m = a.shape[0]//n
    a3D = a.reshape(m, n, -1)
    return a3D[np.random.permutation(m)].reshape(-1,a3D.shape[-1])

逐步运行改组方法 -

1]设置随机输入数组并拆分为3D版本:

In [2]: np.random.seed(0)

In [3]: a = np.random.randint(11,99,(16,3))

In [4]: a3D = a.reshape(a.shape[0]//4,4,-1)

In [5]: a
Out[5]: 
array([[55, 58, 75],
       [78, 78, 20],
       [94, 32, 47],
       [98, 81, 23],
       [69, 76, 50],
       [98, 57, 92],
       [48, 36, 88],
       [83, 20, 31],
       [91, 80, 90],
       [58, 75, 93],
       [60, 40, 30],
       [30, 25, 50],
       [43, 76, 20],
       [68, 43, 42],
       [85, 34, 46],
       [86, 66, 39]])

2]检查3D阵列:

In [6]: a3D
Out[6]: 
array([[[55, 58, 75],
        [78, 78, 20],
        [94, 32, 47],
        [98, 81, 23]],

       [[69, 76, 50],
        [98, 57, 92],
        [48, 36, 88],
        [83, 20, 31]],

       [[91, 80, 90],
        [58, 75, 93],
        [60, 40, 30],
        [30, 25, 50]],

       [[43, 76, 20],
        [68, 43, 42],
        [85, 34, 46],
        [86, 66, 39]]])

3]沿第一轴(原位)随机播放:

In [7]: np.random.shuffle(a3D)

In [8]: a3D
Out[8]: 
array([[[69, 76, 50],
        [98, 57, 92],
        [48, 36, 88],
        [83, 20, 31]],

       [[43, 76, 20],
        [68, 43, 42],
        [85, 34, 46],
        [86, 66, 39]],

       [[55, 58, 75],
        [78, 78, 20],
        [94, 32, 47],
        [98, 81, 23]],

       [[91, 80, 90],
        [58, 75, 93],
        [60, 40, 30],
        [30, 25, 50]]])

4]验证原始数组中的更改:

In [9]: a
Out[9]: 
array([[69, 76, 50],
       [98, 57, 92],
       [48, 36, 88],
       [83, 20, 31],
       [43, 76, 20],
       [68, 43, 42],
       [85, 34, 46],
       [86, 66, 39],
       [55, 58, 75],
       [78, 78, 20],
       [94, 32, 47],
       [98, 81, 23],
       [91, 80, 90],
       [58, 75, 93],
       [60, 40, 30],
       [30, 25, 50]])

运行时测试

In [102]: a = np.random.randint(11,99,(16000,3))

In [103]: df = pd.DataFrame(a)

# @piRSquared's soln1
In [106]: %timeit df.iloc[np.random.permutation(np.arange(df.shape[0]).reshape(-1, 4)).ravel()]
100 loops, best of 3: 2.88 ms per loop

# @piRSquared's soln2
In [107]: %%timeit
     ...: d = df.set_index(np.arange(len(df)) // 4, append=True).swaplevel(0, 1)
     ...: pd.concat([d.xs(i) for i in np.random.permutation(range(4))])
100 loops, best of 3: 3.48 ms per loop

# Array based soln-1
In [108]: %timeit array_shuffle(a, n=4)
100 loops, best of 3: 3.38 ms per loop

# Array based soln-2
In [109]: %timeit array_permuted_indexing(a, n=4)
10000 loops, best of 3: 125 µs per loop

答案 1 :(得分:2)

设置

考虑数据框df

df = pd.DataFrame(np.random.randint(10, size=(16, 4)), columns=list('WXYZ'))
df

    W  X  Y  Z
0   9  8  6  2
1   0  9  5  5
2   7  5  9  4
3   7  1  1  8
4   7  7  2  2
5   5  5  0  2
6   9  3  2  7
7   5  7  2  9
8   6  6  2  8
9   0  7  0  8
10  7  5  5  2
11  6  0  9  5
12  9  2  2  2
13  8  8  2  5
14  4  1  5  6
15  1  2  3  9

选项1
灵感来自@ B.M.和@Divakar
我使用np.random.permutation因为它返回的副本是传递内容的置换版本。这意味着我可以直接将其传递给iloc并返回我需要的内容。

df.iloc[np.random.permutation(np.arange(16).reshape(-1, 4)).ravel()]

    W  X  Y  Z
12  9  2  2  2
13  8  8  2  5
14  4  1  5  6
15  1  2  3  9
0   9  8  6  2
1   0  9  5  5
2   7  5  9  4
3   7  1  1  8
8   6  6  2  8
9   0  7  0  8
10  7  5  5  2
11  6  0  9  5
4   7  7  2  2
5   5  5  0  2
6   9  3  2  7
7   5  7  2  9

选项2

我在添加

时可以调用的索引中添加一个级别
d = df.set_index(np.arange(len(df)) // 4, append=True).swaplevel(0, 1)
d

      W  X  Y  Z
0 0   9  8  6  2
  1   0  9  5  5
  2   7  5  9  4
  3   7  1  1  8
1 4   7  7  2  2
  5   5  5  0  2
  6   9  3  2  7
  7   5  7  2  9
2 8   6  6  2  8
  9   0  7  0  8
  10  7  5  5  2
  11  6  0  9  5
3 12  9  2  2  2
  13  8  8  2  5
  14  4  1  5  6
  15  1  2  3  9

然后我们可以随机播放

pd.concat([d.xs(i) for i in np.random.permutation(range(4))])

    W  X  Y  Z
12  9  2  2  2
13  8  8  2  5
14  4  1  5  6
15  1  2  3  9
4   7  7  2  2
5   5  5  0  2
6   9  3  2  7
7   5  7  2  9
0   9  8  6  2
1   0  9  5  5
2   7  5  9  4
3   7  1  1  8
8   6  6  2  8
9   0  7  0  8
10  7  5  5  2
11  6  0  9  5

答案 2 :(得分:0)

python下面的代码实现了神奇的

from random import shuffle
import numpy as np
from math import ceil

#creating sample dataset
d=[[i*4 +j for i in range(5)] for j in range(25)]
a = np.array(d, int)
print '--------------Input--------------'
print a

gl=4 #group length i.e number of rows needs to be intact
parts=ceil(1.0*len(a)/gl) #no of partitions based on grouplength for the given dataset

#creating partition list and shuffling it to use later 
x = [i for i in range(int(parts))]
shuffle(x)

#Creates new dataset based on shuffled partition list
fg=x.pop(0)
f = a[gl*fg:gl*(fg+1)]
for i in x: 
 t=a[gl*i:(i+1)*gl]
 f=np.concatenate((f, t), axis=0)
print '--------------Output--------------'
print f