Pandas:使用索引

时间:2017-03-30 20:48:01

标签: python pandas dataframe

我正在解析的文本文件包含固定宽度字段,其中的行如下所示:

USC00142401201703TMAX  211  H  133  H  161  H  194  H  206  H  161  H  244  H  178  H-9999     250  H   78  H   44  H   67  H   50  H   39  H  106  H  239  H  239  H  217  H  317  H  311  H  178  H  139  H-9999     228  H-9999   -9999   -9999   -9999   -9999   -9999   

我正在将这些解析为pandas DataFrame,如下所示:

from collections import OrderedDict
from pandas import DataFrame
import pandas as pd
import numpy as np

def read_into_dataframe(station_filepath):

    # specify the fixed-width fields
    column_specs = [(0, 11),   # ID
                    (11, 15),  # year
                    (15, 17),  # month
                    (17, 21),  # variable (referred to as element in the GHCND readme.txt)
                    (21, 26),  # day 1
                    (29, 34),  # day 2
                    (37, 42),  # day 3
                    (45, 50),  # day 4
                    (53, 58),  # day 5
                    (61, 66),  # day 6
                    (69, 74),  # day 7
                    (77, 82),  # day 8
                    (85, 90),  # day 9
                    (93, 98),  # day 10
                    (101, 106),  # day 11
                    (109, 114),  # day 12
                    (117, 122),  # day 13
                    (125, 130),  # day 14
                    (133, 138),  # day 15
                    (141, 146),  # day 16
                    (149, 154),  # day 17
                    (157, 162),  # day 18
                    (165, 170),  # day 19
                    (173, 178),  # day 20
                    (181, 186),  # day 21
                    (189, 194),  # day 22
                    (197, 202),  # day 23
                    (205, 210),  # day 24
                    (213, 218),  # day 25
                    (221, 226),  # day 26
                    (229, 234),  # day 27
                    (237, 242),  # day 28
                    (245, 250),  # day 29
                    (253, 258),  # day 30
                    (261, 266)]  # day 31

    # create column names to correspond with the fields specified above
    column_names = ['station_id', 'year', 'month', 'variable',
                    '01', '02', '03', '04', '05', '06', '07', '08', '09', '10',  
                    '11', '12', '13', '14', '15', '16', '17', '18', '19', '20',  
                    '21', '22', '23', '24', '25', '26', '27', '28', '29', '30',  '31']

    # read the fixed width file into a DataFrame columns with the widths and names specified above
    df = pd.read_fwf(station_filepath, 
                     header=None,
                     colspecs=column_specs,
                     names=column_names,
                     na_values=-9999)

    # convert the variable column to string data type, all others as integer data type
    df.dropna()  #REVISIT do we really want to do this?
    df['variable'] = df['variable'].astype(str)

    # keep only the rows where the variable value is 'PRCP', 'TMIN', or 'TMAX'
    df = df[df['variable'].isin(['PRCP', 'TMAX', 'TMIN'])]

    # melt the individual day columns into a single day column
    df = pd.melt(df,
                 id_vars=['station_id', 'year', 'month', 'variable'],
                 value_vars=['01', '02', '03', '04', '05', '06', '07', '08', '09', '10',
                             '11', '12', '13', '14', '15', '16', '17', '18', '19', '20',
                             '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31'],
                 var_name='day', 
                 value_name='value')

    # pivot the DataFrame on the variable type (PRCP, TMIN, TMAX), so each
    # type has a separate column with the day's value for the type
    df = df.pivot_table(index=['year',
                               'month',
                               'day'],
                        columns='variable',
                        values='value')

    return df

我现在以我想要的形状获取DataFrame,除了有几天不存在的行(即2月31日等),我想删除它。

我尝试使用蒙版进行此操作,但是当我这样做时,当我尝试使用我认为有效的列名时,我得到了一个KeyError。例如,如果我在返回DataFrame之前在上面的函数中包含以下代码,我将得到一个KeyError:

months_with_31days = [1, 3, 7, 8, 10, 12]
df = df[((df['day'] == 31) & (df['month'] in months_with_31days))
        |
       ((df['day'] == 30) & (df['month'] != 2))
        |
       ((df['day'] == 29) & (df['month'] != 2))
        |
       ((df['day'] == 29) & (df['month'] == 2) & calendar.isleap(df['year']))
        | 
        df['day'] < 29]

以上将导致KeyError:

KeyError: 'day'

day变量由melt()调用创建,然后在对pivot_table()的调用中的索引中使用。这对我如何影响DataFrame的索引以及为什么它能够使用以前的列名称的能力尚不清楚。 [编辑]我假设我现在在DatFrame上有一个MultiIndex,它是通过使用索引参数调用pivot_table()而创建的。

打印DataFrame时显示的初始行:

variable         PRCP   TMAX   TMIN
year month day                     
1893 1     01     NaN   61.0   33.0
           02     NaN   33.0    6.0
           03     NaN   44.0   17.0
           04     NaN   78.0   22.0
           05     NaN   17.0  -94.0
           06     NaN   33.0    0.0
           07     NaN    0.0  -67.0

我尝试使用点符号而不是带引号列名的括号来引用DataFrame的列,但是我得到了类似的错误。似乎年,月和日列已合并到单个索引列中,无法再单独引用。或者不是,也许还有其他事情发生在这里?我很难过,也许甚至没有以最好的方式接近这一点,任何帮助或建议都将非常感激。感谢。

1 个答案:

答案 0 :(得分:2)

是的,您已经创建了一个多索引DataFrame。通过查看输出(无法访问您的数据),您应该可以通过键入以下内容来访问日期:

df['variable']['day']