Question

我有一个基于不同气象站的数据集，用于几个变量（温度，压力等），

stationID | Time | Temperature | Pressure |...
----------+------+-------------+----------+
123       |  1   |     30      |  1010.5  |   
123       |  2   |     31      |  1009.0  |
202       |  1   |     24      |  NaN     |
202       |  2   |     24.3    |  NaN     |
202       |  3   |     NaN     |  1000.3  |
...

我想创建一个数据透视表，显示每个气象站的NaN数和非NaN数，例如：

stationID | nanStatus | Temperature | Pressure |...
----------+-----------+-------------+----------+
123       |  NaN      |      0      |     0    |       
          |  nonNaN   |      2      |     2    |
202       |  NaN      |      1      |     2    |
          |  nonNaN   |      2      |     1    |
...

下面我展示了到目前为止我所做的工作，它以温度的方式工作（以繁琐的方式）。但是，如何对两个变量进行相同的处理，如上所示？

import pandas as pd
import bumpy as np
df = pd.DataFrame({'stationID':[123,123,202,202,202], 'Time':[1,2,1,2,3],'Temperature':[30,31,24,24.3,np.nan],'Pressure':[1010.5,1009.0,np.nan,np.nan,1000.3]})

dfnull = df.isnull()
dfnull['stationID'] = df['stationID']
dfnull['tempValue'] = df['Temperature']
dfnull.pivot_table(values=["tempValue"], index=["stationID","Temperature"], aggfunc=len,fill_value=0)

输出结果为：

----------------------------------
                         tempValue
stationID | Temperature           
123       | False                2
202       | False                2
          | True                 1

Answer 1

更新：感谢@root：

In [16]: df.groupby('stationID')[['Temperature','Pressure']].agg([nans, notnans]).astype(int).stack(level=1)
Out[16]:
                   Temperature  Pressure
stationID
123       nans               0         0
          notnans            2         2
202       nans               1         2
          notnans            2         1

原始回答：

In [12]: %paste
def nans(s):
    return s.isnull().sum()

def notnans(s):
    return s.notnull().sum()
## -- End pasted text --

In [37]: df.groupby('stationID')[['Temperature','Pressure']].agg([nans, notnans]).astype(np.int8)
Out[37]:
          Temperature         Pressure
                 nans notnans     nans notnans
stationID
123                 0       2        0       2
202                 1       2        2       1

Answer 2

我承认这不是最漂亮的解决方案，但它确实有效。首先定义两个临时列TempNaN和PresNaN：

df['TempNaN'] = df['Temperature'].apply(lambda x: 'NaN' if x!=x else 'NonNaN')
df['PresNaN'] = df['Pressure'].apply(lambda x: 'NaN' if x!=x else 'NonNaN')

然后使用MultiIndex定义结果DataFrame：

Results = pd.DataFrame(index=pd.MultiIndex.from_tuples(list(zip(*[sorted(list(df['stationID'].unique())*2),['NaN','NonNaN']*df['stationID'].nunique()])),names=['stationID','NaNStatus']))

将您的计算存储在结果DataFrame中：

Results['Temperature'] = df.groupby(['stationID','TempNaN'])['Temperature'].apply(lambda x: x.shape[0])
Results['Pressure'] = df.groupby(['stationID','PresNaN'])['Pressure'].apply(lambda x: x.shape[0])

用零填充空白值：

Results.fillna(value=0,inplace=True)

如果更容易，您可以遍历列。例如：

Results = pd.DataFrame(index=pd.MultiIndex.from_tuples(list(zip(*[sorted(list(df['stationID'].unique())*2),['NaN','NonNaN']*df['stationID'].nunique()])),names=['stationID','NaNStatus']))
for col in ['Temperature','Pressure']:
    df[col + 'NaN'] = df[col].apply(lambda x: 'NaN' if x!=x else 'NonNaN')
    Results[col] = df.groupby(['stationID',col + 'NaN'])[col].apply(lambda x: x.shape[0])
    df.drop([col + 'NaN'],axis=1,inplace=True)
Results.fillna(value=0,inplace=True)

Answer 3

d = {'stationID':[], 'nanStatus':[], 'Temperature':[], 'Pressure':[]}

for station_id, data in df.groupby(['stationID']):

    temp_nans = data.isnull().Temperature.mean()*data.isnull().Temperature.count()
    pres_nans = data.isnull().Pressure.mean()*data.isnull().Pressure.count()

    d['stationID'].append(station_id)
    d['nanStatus'].append('NaN')
    d['Temperature'].append(temp_nans)
    d['Pressure'].append(pres_nans)

    d['stationID'].append(station_id)
    d['nanStatus'].append('nonNaN')
    d['Temperature'].append(data.isnull().Temperature.count() - temp_nans)
    d['Pressure'].append(data.isnull().Pressure.count() - pres_nans)

df2 = pd.DataFrame.from_dict(d)
print(df2)

结果是：

   Pressure  Temperature nanStatus  stationID
0       0.0          0.0       NaN        123
1       2.0          2.0    nonNaN        123
2       2.0          1.0       NaN        202
3       1.0          2.0    nonNaN        202

Python pandas - 构建多变量数据透视表以显示NaN和非NaN的数量

3 个答案: