我正在解析的文本文件包含固定宽度字段,其中的行如下所示:
USC00142401201703TMAX 211 H 133 H 161 H 194 H 206 H 161 H 244 H 178 H-9999 250 H 78 H 44 H 67 H 50 H 39 H 106 H 239 H 239 H 217 H 317 H 311 H 178 H 139 H-9999 228 H-9999 -9999 -9999 -9999 -9999 -9999
我正在将这些解析为pandas DataFrame,如下所示:
from collections import OrderedDict
from pandas import DataFrame
import pandas as pd
import numpy as np
def read_into_dataframe(station_filepath):
# specify the fixed-width fields
column_specs = [(0, 11), # ID
(11, 15), # year
(15, 17), # month
(17, 21), # variable (referred to as element in the GHCND readme.txt)
(21, 26), # day 1
(29, 34), # day 2
(37, 42), # day 3
(45, 50), # day 4
(53, 58), # day 5
(61, 66), # day 6
(69, 74), # day 7
(77, 82), # day 8
(85, 90), # day 9
(93, 98), # day 10
(101, 106), # day 11
(109, 114), # day 12
(117, 122), # day 13
(125, 130), # day 14
(133, 138), # day 15
(141, 146), # day 16
(149, 154), # day 17
(157, 162), # day 18
(165, 170), # day 19
(173, 178), # day 20
(181, 186), # day 21
(189, 194), # day 22
(197, 202), # day 23
(205, 210), # day 24
(213, 218), # day 25
(221, 226), # day 26
(229, 234), # day 27
(237, 242), # day 28
(245, 250), # day 29
(253, 258), # day 30
(261, 266)] # day 31
# create column names to correspond with the fields specified above
column_names = ['station_id', 'year', 'month', 'variable',
'01', '02', '03', '04', '05', '06', '07', '08', '09', '10',
'11', '12', '13', '14', '15', '16', '17', '18', '19', '20',
'21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31']
# read the fixed width file into a DataFrame columns with the widths and names specified above
df = pd.read_fwf(station_filepath,
header=None,
colspecs=column_specs,
names=column_names,
na_values=-9999)
# convert the variable column to string data type, all others as integer data type
df.dropna() #REVISIT do we really want to do this?
df['variable'] = df['variable'].astype(str)
# keep only the rows where the variable value is 'PRCP', 'TMIN', or 'TMAX'
df = df[df['variable'].isin(['PRCP', 'TMAX', 'TMIN'])]
# melt the individual day columns into a single day column
df = pd.melt(df,
id_vars=['station_id', 'year', 'month', 'variable'],
value_vars=['01', '02', '03', '04', '05', '06', '07', '08', '09', '10',
'11', '12', '13', '14', '15', '16', '17', '18', '19', '20',
'21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31'],
var_name='day',
value_name='value')
# pivot the DataFrame on the variable type (PRCP, TMIN, TMAX), so each
# type has a separate column with the day's value for the type
df = df.pivot_table(index=['year',
'month',
'day'],
columns='variable',
values='value')
return df
我现在以我想要的形状获取DataFrame,除了有几天不存在的行(即2月31日等),我想删除它。
我尝试使用蒙版进行此操作,但是当我这样做时,当我尝试使用我认为有效的列名时,我得到了一个KeyError。例如,如果我在返回DataFrame之前在上面的函数中包含以下代码,我将得到一个KeyError:
months_with_31days = [1, 3, 7, 8, 10, 12]
df = df[((df['day'] == 31) & (df['month'] in months_with_31days))
|
((df['day'] == 30) & (df['month'] != 2))
|
((df['day'] == 29) & (df['month'] != 2))
|
((df['day'] == 29) & (df['month'] == 2) & calendar.isleap(df['year']))
|
df['day'] < 29]
以上将导致KeyError:
KeyError: 'day'
day变量由melt()调用创建,然后在对pivot_table()的调用中的索引中使用。这对我如何影响DataFrame的索引以及为什么它能够使用以前的列名称的能力尚不清楚。 [编辑]我假设我现在在DatFrame上有一个MultiIndex,它是通过使用索引参数调用pivot_table()而创建的。
打印DataFrame时显示的初始行:
variable PRCP TMAX TMIN
year month day
1893 1 01 NaN 61.0 33.0
02 NaN 33.0 6.0
03 NaN 44.0 17.0
04 NaN 78.0 22.0
05 NaN 17.0 -94.0
06 NaN 33.0 0.0
07 NaN 0.0 -67.0
我尝试使用点符号而不是带引号列名的括号来引用DataFrame的列,但是我得到了类似的错误。似乎年,月和日列已合并到单个索引列中,无法再单独引用。或者不是,也许还有其他事情发生在这里?我很难过,也许甚至没有以最好的方式接近这一点,任何帮助或建议都将非常感激。感谢。
答案 0 :(得分:2)
是的,您已经创建了一个多索引DataFrame。通过查看输出(无法访问您的数据),您应该可以通过键入以下内容来访问日期:
df['variable']['day']