使用NCML减少netCDF文件中的维度

时间:2013-09-17 18:24:29

标签: netcdf ncml thredds

我正在尝试使用NCML将CF-1.4文件“转换”为CF-1.6。特别感兴趣的是如何1)去除尺寸然后2)改变变量的尺寸。例如,下面是两个ncdump文件的顶部(netCDF)。第一个是CF-1.4,维度为timezlatlon。在此文件中,变量(例如,temp)是这四个函数:temp(time,z,lat,lon)。我想通过NCML将其转换为CF-1.6文件,如第二个文件所示,其中z / lat / lon不再是维度,变量是函数只有时间。谢谢,

文件1:

netcdf wqb_1.4 {
dimensions:
        time = UNLIMITED ; // (109008 currently)
        z = 1 ;
        lat = 1 ;
        lon = 1 ;
variables:
        float time(time) ;
                time:long_name = "Time" ;
                time:standard_name = "time" ;
                time:short_name = "time" ;
                time:axis = "T" ;
                time:units = "minutes since 2008-01-01 00:00:00 -10:00" ;
        float z(z) ;
                z:long_name = "depth below mean sea level" ;
                z:standard_name = "depth" ;
                z:short_name = "depth" ;
                z:axis = "z" ;
                z:units = "meters" ;
        float lat(lat) ;
                lat:long_name = "Latitude" ;
                lat:standard_name = "latitude" ;
                lat:short_name = "lat" ;
                lat:axis = "Y" ;
                lat:units = "degrees_north" ;
        float lon(lon) ;
                lon:long_name = "Longitude" ;
                lon:standard_name = "longitude" ;
                lon:short_name = "lon" ;
                lon:axis = "X" ;
                lon:units = "degrees_east" ;
        float temp(time, z, lat, lon) ;
                temp:long_name = "Temperature" ;
                temp:standard_name = "sea_water_temperature" ;
                temp:short_name = "temp" ;
                temp:units = "Celsius" ;
                temp:coordinates = "time lat lon alt" ;
                temp:valid_range = 10., 35. ;
                temp:_FillValue = -999.f ;
                temp:observation_type = "measured" ;

文件2:

netcdf wqb_1.6 {
dimensions:
        time = UNLIMITED ; // (109008 currently)
        name_strlen = 5 ;
variables:
        char station_name(name_strlen) ;
                station_name:long_name = "wqbaw" ;
                station_name:cf_role = "timeseries_id" ;
        float time(time) ;
                time:long_name = "Time" ;
                time:standard_name = "time" ;
                time:short_name = "time" ;
                time:axis = "T" ;
                time:units = "minutes since 2008-01-01 00:00:00 -10:00" ;
        float z ;
                z:long_name = "depth below mean sea level" ;
                z:standard_name = "depth" ;
                z:short_name = "depth" ;
                z:axis = "z" ;
                z:units = "meters" ;
        float lat ;
                lat:long_name = "Latitude" ;
                lat:standard_name = "latitude" ;
                lat:short_name = "lat" ;
                lat:axis = "Y" ;
                lat:units = "degrees_north" ;
        float lon ;
                lon:long_name = "Longitude" ;
                lon:standard_name = "longitude" ;
                lon:short_name = "lon" ;
                lon:axis = "X" ;
                lon:units = "degrees_east" ;
        float temp(time) ;
                temp:long_name = "Temperature" ;
                temp:standard_name = "sea_water_temperature" ;
                temp:short_name = "temp" ;
                temp:units = "Celsius" ;
                temp:coordinates = "time lat lon alt" ;
                temp:valid_range = 10., 35. ;
                temp:_FillValue = -999.f ;
                temp:observation_type = "measured" ;

5 个答案:

答案 0 :(得分:3)

<强>更新下面的解决方案可以正常工作,但:从中提取数据失败,正如John M.发现的那样(参见其他答案) )。我们认为我们已经发现维持单个维度是解决方案,但从四个维度到一个维度最终会导致错误。正如Sean A.所指出的,你无法使用NcML改变变量的形状。

原创“解决方案”(实际上并不奏效):

如果您的目标是使您的数据符合CF-1.6,则可以将该维度设为station,其值为1。所以你可以这样做:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/usgs/data/file1.nc">
  <remove type="dimension" name="lon"/>
  <remove type="dimension" name="lat"/>
  <remove type="dimension" name="z"/>
  <dimension name="station" length="1"/>
  <dimension name="name_strlen" length="20" />
  <variable name="lat" shape="station"/>
  <variable name="lon" shape="station"/>
  <variable name="z" shape="station"/>
  <variable name="temp" shape="time station"/>
  <variable name="site" shape="station name_strlen" type="char">
    <attribute name="standard_name" value="station_id" />
    <attribute name="cf_role" value="timeseries_id" />
    <values> my_station_001 </values>
  </variable>
  <attribute name="Conventions" value="CF-1.6" />
  <attribute name="featureType" value="timeSeries" />
</netcdf>

答案 1 :(得分:3)

Rich的解决方案将针对这个非常具体的案例进行处理,但出于错误的原因。在NcML中,您可以删除维对象,但不能重新整形数据变量。对于这种特殊情况,当您尝试删除单个维度(大小为1)时,事情似乎会解决,因为它并没有真正改变数据在磁盘上的布局方式。例如,如果您使用Unidata的toolsUI使用Rich的答案中的NcML为temp变量执行ncdump,您将看到单例维度仍然存在且尚未真正删除。我不确定这会如何影响您的文件读取 - 我认为这取决于客户端。但是,如果您尝试删除非singelton维度,那么这将失败。

如果您确实想要正确地重塑数据,则必须重写netCDF文件。不幸的是,我所知道的并没有任何“捷径”。例如,如果在Unidata的toolsUI中使用Rich的NcML并尝试基于它写出新文件,则会出现错误,例如“ERROR:section(1)中的范围数量必须为= 0” Z“。这是因为netCDF文件中仍然存在单例维度,但是NcML文件试图强制范围为0.但是,如果你知道python,那么写一个脚本来重写你的netCDF文件应该非常简单。

请注意,使用NcML重塑变量的功能是我们定期听到的功能请求 - 将功能请求发送到support-netcdf-java@unidata.ucar.edu并不会有什么坏处。另请注意,Unidata是一个社区驱动的组织,Rich将在我们的用户委员会上,该委员会将在下个月召开会议。我建议他也在会议上提到功能请求。

干杯!

肖恩

答案 2 :(得分:2)

Rich是正确的,这样做的目的是尝试将我们的数据提升到CF-1.6,这样做的目的是让我们通过SOS提供数据。更具体地说,我们想使用ncSOS(基于TDS构建),这种特殊的SOS风格需要CF-1.6。在这方面,通过NCML的修改似乎有效(加上一些额外的修改;见下文)。

我不想修改数据集,其中一些可以追溯到几年前。 Sean关于客户端工具的观点也很重要,因为我们的许多用例都涉及需要变量维度为lat / lon的工具。因此,我们的解决方案是通过两个NCML'包装器'通过TDS提供一个数据集,一个用于ncSOS,另一个用于通过OPeNDAP访问需要lat / lon的特定客户端。

除了上面Rich的建议,为了在ncSOS中工作,我们不得不:

  1. 添加CF-1.6'坐标'属性(“time lat lon z”)
  2. 添加全局属性“featureType = timeSeries”
  3. 添加station_name变量
  4. 从点到站更改数据类型
  5. 结果如下:

        <attribute name="featureType" value="timeSeries" />
        <remove type="dimension" name="lon"/>
        <remove type="dimension" name="lat"/>
        <remove type="dimension" name="z"/>
        <dimension name="name_strlen" length="4"/>
        <variable name="lat" shape=""/>
        <variable name="lon" shape=""/>
        <variable name="z" shape=""/>
        <variable name="station_name" shape="name_strlen" type="char">
          <attribute name="long_name" value="NS01" />
          <attribute name="cf_role" value="timeseries_id" />
          <values>NS01</values>
        </variable>
        <variable name="temp" shape="time">
          <attribute name="coordinates" value="time lat lon z" />
        </variable>
    

答案 3 :(得分:2)

为了跟进Jim上面的帖子,虽然Rich的NcML解决方案似乎最初起作用,但尝试通过OPeNDAP或ncSOS获取数据是不成功的,证实了Sean对此的怀疑。

目录显示成功,OPeNDAP表单显示CF-1.6的新维度和重新整形变量。此外,ncSOS GetCapabilities文档也会成功显示。

但是,尝试使用OPeNDAP表单下载某些数据存在问题。我无法在OPeNDAP表单上获得变量的子集。例如:

http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.html

如果我尝试使用此URL获取第一个临时值:

http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.ascii?temp [0:1:0]

它给了我这个错误:

Error {
    code = 500;
    message = "NcSDArray InvalidRangeException=Number of ranges in section (1) must be = 4";
};

唯一成功的是获得所有价值观:

http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.ascii?temp [0:1:359]

此外,尝试通过ncSOS GetObservation获取数据也会失败。尝试了以下URL:

http://oos.soest.hawaii.edu/thredds-test/sos/hioos/nss/ns01agg.ncml?service=SOS&version=1.0.0&request=GetObservation&responseFormat=text%2Fxml%3Bsubtype%3D%22om%2F1.0.0%22&offering=NS01&observedProperty=temp&procedure=urn:ioos:station:org.pacioos:NS01

这导致threddsServlet.log中出现以下错误消息:

2013-10-02T09:03:44.844 -1000 [1288472818][    2602] INFO  - threddsServlet - Remote host: 128.171.151.240 - Request: "GET /thredds-test/sos/hioos/nss/ns01agg.ncml?service=SOS&version=1.0.0&request=GetObs
ervation&responseFormat=text%2Fxml%3Bsubtype%3D%22om%2F1.0.0%22&offering=NS01&observedProperty=temp&procedure=urn:ioos:station:org.pacioos:NS01 HTTP/1.1"
2013-10-02T09:03:44.845 -1000 [1288472819][    2602] INFO  - com.asascience.ncsos.controller.SosController - Handling SOS metadata request.
2013-10-02T09:03:45.614 -1000 [1288473588][    2602] ERROR - ucar.nc2.Structure - Structure.IteratorRank1.readNext()
ucar.ma2.InvalidRangeException: Number of ranges in section (1) must be = 4
    at ucar.ma2.Section.fill(Section.java:144)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.Variable.read(Variable.java:647)
    at ucar.nc2.ncml.AggregationOuterDimension$DatasetOuterDimension.read(AggregationOuterDimension.java:774)
    at ucar.nc2.ncml.AggregationOuterDimension.reallyRead(AggregationOuterDimension.java:293)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:553)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.Variable.read(Variable.java:647)
    at ucar.nc2.dataset.StructurePseudoDS.reallyRead(StructurePseudoDS.java:193)
    at ucar.nc2.Variable._read(Variable.java:861)
    at ucar.nc2.Variable.read(Variable.java:673)
    at ucar.nc2.Variable.read(Variable.java:619)
    at ucar.nc2.Structure.readStructure(Structure.java:378)
    at ucar.nc2.Structure$IteratorRank1.readNext(Structure.java:464)
    at ucar.nc2.Structure$IteratorRank1.next(Structure.java:447)
    at ucar.nc2.ft.point.PointIteratorFromStructureData.nextStructureData(PointIteratorFromStructureData.java:103)
    at ucar.nc2.ft.point.PointIteratorFromStructureData.hasNext(PointIteratorFromStructureData.java:68)
    at ucar.nc2.ft.point.PointCollectionImpl.calcBounds(PointCollectionImpl.java:128)
    at com.asascience.ncsos.util.DatasetHandlerAdapter.calcBounds(DatasetHandlerAdapter.java:122)
    at com.asascience.ncsos.cdmclasses.TimeSeries.setData(TimeSeries.java:243)
    at com.asascience.ncsos.getobs.SOSGetObservationRequestHandler.setCDMDatasetForStations(SOSGetObservationRequestHandler.java:193)
    at com.asascience.ncsos.getobs.SOSGetObservationRequestHandler.<init>(SOSGetObservationRequestHandler.java:138)
    at com.asascience.ncsos.service.SOSParser.enhanceGETRequest(SOSParser.java:197)
    at com.asascience.ncsos.controller.SosController.handleSOSRequest(SosController.java:80)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:102)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:48)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
2013-10-02T09:03:45.616 -1000 [1288473590][    2602] ERROR - com.asascience.ncsos.util.DatasetHandlerAdapter - Could not calculate the bounds of the PointFeatureCollection NS01
Structure.Iterator.readNext()
2013-10-02T09:03:45.616 -1000 [1288473590][    2602] ERROR - com.asascience.ncsos.cdmclasses.baseCDMClass - TimeSeries - setData; exception:
java.lang.NullPointerException
2013-10-02T09:03:45.616 -1000 [1288473590][    2602] ERROR - com.asascience.ncsos.service.SOSParser - java.lang.NullPointerException
2013-10-02T09:03:45.617 -1000 [1288473591][    2602] ERROR - com.asascience.ncsos.controller.SosController -
2013-10-02T09:03:45.817 -1000 [1288473791][    2602] INFO  - threddsServlet - Request Completed - 200 - -1 - 973:1

答案 4 :(得分:2)

NcML now(自4.4版开始)具有删除长度为1的维度的操作,例如:

<variable name="temp">
  <logicalReduce dimNames="lat lon" />
</variable>

http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ncml/AnnotatedSchema4.html#logicalReduce