Question

我有一个500 * 100的Pandas DataFrame，有很多NaN值。我知道每列将包含495个NaN和5个实数值。我想重塑表格，只包含5行的实际值，因此最终尺寸应为5 * 100.

我知道有很多关于如何删除NaN值的问题，但我还没有找到一种方法来相应地重塑表格。

提前致谢。

Answer 1

apply需要numpy array，只需创建Series并重置df.apply(lambda x: pd.Series(x.dropna().values))以重置索引：

df = pd.DataFrame({'B':[4,np.nan,4,np.nan,np.nan,4],
                   'C':[7,np.nan,9,np.nan,2,np.nan],
                   'D':[1,3,np.nan,7,np.nan,np.nan],
                   'E':[np.nan,3,np.nan,9,2,np.nan]})

print (df)
     B    C    D    E
0  4.0  7.0  1.0  NaN
1  NaN  NaN  3.0  3.0
2  4.0  9.0  NaN  NaN
3  NaN  NaN  7.0  9.0
4  NaN  2.0  NaN  2.0
5  4.0  NaN  NaN  NaN

df1 = df.apply(lambda x: pd.Series(x.dropna().values))
print (df1)
     B    C    D    E
0  4.0  7.0  1.0  3.0
1  4.0  9.0  3.0  9.0
2  4.0  2.0  7.0  2.0

样品：

<activity
        android:name=".MainActivity"
        android:label="@string/app_name"
        android:theme="@style/AppTheme.NoActionBar">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
    </activity>

Answer 2

方法＃1 这是一个有数组数据的人 -

a = df.values.T
df_out = pd.DataFrame(a[~np.isnan(a)].reshape(a.shape[0],-1).T)

示例运行 -

In [450]: df
Out[450]: 
     0    1    2
0  1.0  NaN  NaN
1  9.0  7.0  8.0
2  NaN  NaN  NaN
3  NaN  5.0  7.0

In [451]: a = df.values.T

In [452]: pd.DataFrame(a[~np.isnan(a)].reshape(a.shape[0],-1).T)
Out[452]: 
     0    1    2
0  1.0  7.0  8.0
1  9.0  5.0  7.0

方法＃2 事实证明，我们已经有了一个实用工具：justify -

In [1]: df
Out[1]: 
     0    1    2
0  1.0  NaN  NaN
1  9.0  7.0  8.0
2  NaN  NaN  NaN
3  NaN  5.0  7.0

In [2]: pd.DataFrame(justify(df.values, invalid_val=np.nan, axis=0, side='up')[:2])
Out[2]: 
     0    1    2
0  1.0  7.0  8.0
1  9.0  5.0  7.0

基准

方法 -

def app0(df): # @jezrael's soln
    return df.apply(lambda x: pd.Series(x.dropna().values))

def app1(df): # Proposed in this post
    a = df.values.T
    return pd.DataFrame(a[~np.isnan(a)].reshape(a.shape[0],-1).T)

def app2(df): # Proposed in this post
    a = df.values
    return pd.DataFrame(justify(a, invalid_val=np.nan, axis=0, side='up')[:5])

def app3(df): # @piRSquared's soln-1
    v = df.values
    r = np.arange(v.shape[1])[None, :]
    a = np.isnan(v).argsort(0)
    return pd.DataFrame(v[a[:5], r], columns=df.columns)

def app4(df): # @piRSquared's soln-2
    return pd.DataFrame(
        (lambda a, s: a[~np.isnan(a)].reshape(-1, s, order='F'))
        (df.values.ravel('F'), df.shape[1]),
        columns=df.columns
    )

计时 -

In [513]: # Setup input dataframe with exactly 5 non-NaNs per col
     ...: m,n = 500,100
     ...: N = 5
     ...: a = np.full((m,n), np.nan)
     ...: row_idx = np.random.rand(m,n).argsort(0)[:N]
     ...: a[row_idx, np.arange(n)] = np.random.randint(0,9,(N,n))
     ...: df = pd.DataFrame(a)
     ...: 

In [572]: %timeit app0(df)
     ...: %timeit app1(df)
     ...: %timeit app2(df)
     ...: %timeit app3(df)
     ...: %timeit app4(df)
     ...: 
10 loops, best of 3: 46.1 ms per loop
10000 loops, best of 3: 132 µs per loop
1000 loops, best of 3: 554 µs per loop
1000 loops, best of 3: 446 µs per loop
10000 loops, best of 3: 148 µs per loop

Answer 3

使用@Divakar的示例数据框

df

     0    1    2
0  1.0  NaN  NaN
1  9.0  7.0  8.0
2  NaN  NaN  NaN
3  NaN  5.0  7.0

v = df.values
r = np.arange(v.shape[1])[None, :]
a = np.isnan(v).argsort(0)

pd.DataFrame(v[a[:2], r], columns=df.columns)

     0    1    2
0  1.0  7.0  8.0
1  9.0  5.0  7.0

受@Divakar的回答启发

pd.DataFrame(
    (lambda a, s: a[~np.isnan(a)].reshape(-1, s, order='F'))(df.values.ravel('F'), df.shape[1]),
    columns=df.columns
)

     0    1    2
0  1.0  7.0  8.0
1  9.0  5.0  7.0

从pandas dataframe和reshape表中删除NaN值

3 个答案:

基准