我正在使用xarray读取一个相当小的NetCDF文件（5.4MB），并希望将其转换为Pandas数据框：
import xarray as xr
f = xr.open_dataset('file.nc')
到目前为止，Python使用的内存很少（〜75 MB），但是一旦我调用：
f2 = f.to_dataframe()
内存使用量激增（> 12 GB）。有谁知道为什么会这样吗？我尝试使用to_dask_dataframe()，但在NetCDF文件中的某些数据类型上却出现了错误。
我上传了NetCDF文件here，原始数据被随机数替换，因为我无法分发原始数据。
根据评论的要求：
In [3]: f
Out[3]:
<xarray.Dataset>
Dimensions: (day_in_time_interval: 3652, nv: 2, time: 175296)
Coordinates:
* time (time) datetime64[ns] 2001-01-01 2001-01-01T00:30:00 ...
Dimensions without coordinates: day_in_time_interval, nv
Data variables:
iso_dataset |S1 ...
product |S1 ...
station_details |S1 ...
date (time) int32 ...
valid_dates (day_in_time_interval) int8 ...
time_bnds (time, nv) float32 ...
C020 (time) float32 ...
C060 (time) float32 ...
C120 (time) float32 ...
C200 (time) float32 ...
Attributes:
institution: Royal Netherlands Meteorological Institute (KNMI)
comment: none
Conventions: CF-1.4
location: CESAR observatory, the Netherlands
file_creation_date_time: 20161130 09:34:56 (UTC)
原始文件的ncdump：
netcdf ecnco2 {
dimensions:
time = UNLIMITED ; // (175296 currently)
nv = 2 ;
day_in_time_interval = 3652 ;
variables:
char iso_dataset ;
iso_dataset:hierarchyLevel = "dataset" ;
iso_dataset:url = "http://www.cesar-database.nl" ;
iso_dataset:protocol = "website" ;
iso_dataset:topic = "climatologyMeteorologyAtmosphere" ;
iso_dataset:westbound_longitude = "4.926" ;
iso_dataset:eastbound_longitude = "4.926" ;
iso_dataset:southbound_latitude = "51.97" ;
iso_dataset:northbound_latitude = "51.97" ;
iso_dataset:datasetDateType = "publication" ;
iso_dataset:code = "28992" ;
iso_dataset:codeSpace = "EPSG" ;
iso_dataset:accessConstraints = "CESAR data policy" ;
iso_dataset:useLimitation = "None" ;
iso_dataset:organisationName_dataset = "Royal Netherlands Meteorological Institute (KNMI)" ;
iso_dataset:email_dataset = "fred.bosveld@knmi.nl" ;
iso_dataset:role_dataset = "Principle Investigator" ;
iso_dataset:organisationName_metadata = "Royal Netherlands Meteorological Institute (KNMI)" ;
iso_dataset:role_metadata = "Principle Investigator" ;
iso_dataset:email_metadata = "fred.bosveld@knmi.nl" ;
iso_dataset:url_metadata = "http://www.knmi.nl/~bosveld" ;
iso_dataset:metadataDateType = "creation" ;
iso_dataset:language = "eng" ;
iso_dataset:metadataStandardName = "ISO-19115" ;
iso_dataset:metadataStandardNameVersion = "Nederlands profiel op ISO 19115 voor geografie, v1.2" ;
char product ;
product:format_version = "netCDF,3.6" ;
product:originator = "Bosveld, F.C., KNMI" ;
product:software_version = "see http://www.knmi.nl/~bosveld -> software -> Mobibase" ;
product:command_line = " ncselect.x ecnco2 a30 [M]cesar,[o]ecnco2 2001,2010 -fecnco2.nc" ;
product:date_start_of_data = "2001-01-01T00:00Z" ;
product:date_end_of_data = "2010-12-31T23:59Z" ;
product:revision_date = "2016-11-30" ;
char station_details ;
station_details:name = "CESAR observatory" ;
station_details:latitude = "51.97" ;
station_details:longitude = "4.926" ;
station_details:elevation = "-0.7" ;
station_details:WMO_id = "06348" ;
station_details:address = "Zijdeweg 1" ;
station_details:postal_code = "3411 MH" ;
station_details:city = "Lopik" ;
station_details:administration_area = "Utrecht" ;
station_details:country = "the Netherlands" ;
float time(time) ;
time:units = "hours since 2001-01-01 00:00:00 0:00" ;
time:long_name = "hours since 2001-01-01 00:00:00 (UTC)" ;
time:standard_name = "time" ;
time:axis = "T" ;
time:bounds = "time_bnds" ;
int date(time) ;
date:long_name = "yyyymmdd" ;
byte valid_dates(day_in_time_interval) ;
valid_dates:comment = "indicates whether any data are included for a particular day: 0=none, 1=data, index runs from date indicated by \"units\" attribute of the time variable" ;
float time_bnds(time, nv) ;
float C020(time) ;
C020:units = "ppm" ;
C020:long_name = "CO2 concentration ECN at 20 m" ;
C020:_FillValue = -9999.f ;
C020:cell_methods = "time: mean" ;
float C060(time) ;
C060:units = "ppm" ;
C060:long_name = "CO2 concentration ECN at 60 m" ;
C060:_FillValue = -9999.f ;
C060:cell_methods = "time: mean" ;
float C120(time) ;
C120:units = "ppm" ;
C120:long_name = "CO2 concentration ECN at 120 m" ;
C120:_FillValue = -9999.f ;
C120:cell_methods = "time: mean" ;
float C200(time) ;
C200:units = "ppm" ;
C200:long_name = "CO2 concentration ECN at 200 m" ;
C200:_FillValue = -9999.f ;
C200:cell_methods = "time: mean" ;
// global attributes:
:institution = "Royal Netherlands Meteorological Institute (KNMI)" ;
:comment = "none" ;
:Conventions = "CF-1.4" ;
:location = "CESAR observatory, the Netherlands" ;
:file_creation_date_time = "20161130 09:34:56 (UTC)" ;
:_Format = "classic" ;
}
答案 0 :(得分：2)
之所以发生这种情况，是因为您的数据集具有多个维度，并且要在一个数据帧中表示所有这些维度，则必须进行大量广播。为了说明这一点，让我们减小数据集的大小：
In [8]: ds_small = ds.isel(time=slice(0, 4), day_in_time_interval=slice(0, 2))
In [9]: ds_small
Out[9]:
<xarray.Dataset>
Dimensions: (day_in_time_interval: 2, nv: 2, time: 4)
Coordinates:
* time (time) datetime64[ns] 2001-01-01 2001-01-01T00:30:00 ...
Dimensions without coordinates: day_in_time_interval, nv
Data variables:
iso_dataset |S1 ...
product |S1 ...
station_details |S1 ...
date (time) int32 ...
valid_dates (day_in_time_interval) int8 ...
time_bnds (time, nv) float32 ...
C020 (time) float32 ...
C060 (time) float32 ...
C120 (time) float32 ...
C200 (time) float32 ...
Attributes:
institution: Royal Netherlands Meteorological Institute (KNMI)
comment: none
Conventions: CF-1.4
location: CESAR observatory, the Netherlands
file_creation_date_time: 20161130 09:34:56 (UTC)
In [10]: ds_small.to_dataframe()
Out[10]:
iso_dataset product station_details date valid_dates time_bnds C020 C060 C120 C200
day_in_time_interval nv time
0 0 2001-01-01 00:00:00 b'' b'' b'' 20010101 0 0.0 0.749853 0.311870 0.644066 0.231409
2001-01-01 00:30:00 b'' b'' b'' 20010101 0 0.5 0.758620 0.948448 0.089245 0.632072
2001-01-01 01:00:00 b'' b'' b'' 20010101 0 1.0 0.649947 0.542748 0.422275 0.555378
2001-01-01 01:30:00 b'' b'' b'' 20010101 0 1.5 0.972251 0.766816 0.180199 0.441256
1 2001-01-01 00:00:00 b'' b'' b'' 20010101 0 0.5 0.749853 0.311870 0.644066 0.231409
2001-01-01 00:30:00 b'' b'' b'' 20010101 0 1.0 0.758620 0.948448 0.089245 0.632072
2001-01-01 01:00:00 b'' b'' b'' 20010101 0 1.5 0.649947 0.542748 0.422275 0.555378
2001-01-01 01:30:00 b'' b'' b'' 20010101 0 2.0 0.972251 0.766816 0.180199 0.441256
1 0 2001-01-01 00:00:00 b'' b'' b'' 20010101 0 0.0 0.749853 0.311870 0.644066 0.231409
2001-01-01 00:30:00 b'' b'' b'' 20010101 0 0.5 0.758620 0.948448 0.089245 0.632072
2001-01-01 01:00:00 b'' b'' b'' 20010101 0 1.0 0.649947 0.542748 0.422275 0.555378
2001-01-01 01:30:00 b'' b'' b'' 20010101 0 1.5 0.972251 0.766816 0.180199 0.441256
1 2001-01-01 00:00:00 b'' b'' b'' 20010101 0 0.5 0.749853 0.311870 0.644066 0.231409
2001-01-01 00:30:00 b'' b'' b'' 20010101 0 1.0 0.758620 0.948448 0.089245 0.632072
2001-01-01 01:00:00 b'' b'' b'' 20010101 0 1.5 0.649947 0.542748 0.422275 0.555378
2001-01-01 01:30:00 b'' b'' b'' 20010101 0 2.0 0.972251 0.766816 0.180199 0.441256
因此，您的数据集3维最终产生了一个3级MultiIndex，其长度等于本例中维（4 * 2 * 2）的乘积。
您可能想要的是通过删除一些虚假变量/维度来稍微清理数据集。以下行适用于您共享的小型或完整数据集：
In [14]: ds[['C020', 'C060', 'C120', 'C200']].to_dataframe().head()
Out[14]:
C020 C060 C120 C200
time
2001-01-01 00:00:00 0.749853 0.311870 0.644066 0.231409
2001-01-01 00:30:00 0.758620 0.948448 0.089245 0.632072
2001-01-01 01:00:00 0.649947 0.542748 0.422275 0.555378
2001-01-01 01:30:00 0.972251 0.766816 0.180199 0.441256
2001-01-01 02:00:00 0.302412 0.389957 0.702632 0.497293