我正在尝试使用NCML将CF-1.4文件“转换”为CF-1.6。特别感兴趣的是如何1)去除尺寸然后2)改变变量的尺寸。例如,下面是两个ncdump
文件的顶部(netCDF
)。第一个是CF-1.4,维度为time
,z
,lat
和lon
。在此文件中,变量(例如,temp)是这四个函数:temp(time,z,lat,lon)
。我想通过NCML将其转换为CF-1.6文件,如第二个文件所示,其中z
/ lat
/ lon
不再是维度,变量是函数只有时间。谢谢,
文件1:
netcdf wqb_1.4 {
dimensions:
time = UNLIMITED ; // (109008 currently)
z = 1 ;
lat = 1 ;
lon = 1 ;
variables:
float time(time) ;
time:long_name = "Time" ;
time:standard_name = "time" ;
time:short_name = "time" ;
time:axis = "T" ;
time:units = "minutes since 2008-01-01 00:00:00 -10:00" ;
float z(z) ;
z:long_name = "depth below mean sea level" ;
z:standard_name = "depth" ;
z:short_name = "depth" ;
z:axis = "z" ;
z:units = "meters" ;
float lat(lat) ;
lat:long_name = "Latitude" ;
lat:standard_name = "latitude" ;
lat:short_name = "lat" ;
lat:axis = "Y" ;
lat:units = "degrees_north" ;
float lon(lon) ;
lon:long_name = "Longitude" ;
lon:standard_name = "longitude" ;
lon:short_name = "lon" ;
lon:axis = "X" ;
lon:units = "degrees_east" ;
float temp(time, z, lat, lon) ;
temp:long_name = "Temperature" ;
temp:standard_name = "sea_water_temperature" ;
temp:short_name = "temp" ;
temp:units = "Celsius" ;
temp:coordinates = "time lat lon alt" ;
temp:valid_range = 10., 35. ;
temp:_FillValue = -999.f ;
temp:observation_type = "measured" ;
文件2:
netcdf wqb_1.6 {
dimensions:
time = UNLIMITED ; // (109008 currently)
name_strlen = 5 ;
variables:
char station_name(name_strlen) ;
station_name:long_name = "wqbaw" ;
station_name:cf_role = "timeseries_id" ;
float time(time) ;
time:long_name = "Time" ;
time:standard_name = "time" ;
time:short_name = "time" ;
time:axis = "T" ;
time:units = "minutes since 2008-01-01 00:00:00 -10:00" ;
float z ;
z:long_name = "depth below mean sea level" ;
z:standard_name = "depth" ;
z:short_name = "depth" ;
z:axis = "z" ;
z:units = "meters" ;
float lat ;
lat:long_name = "Latitude" ;
lat:standard_name = "latitude" ;
lat:short_name = "lat" ;
lat:axis = "Y" ;
lat:units = "degrees_north" ;
float lon ;
lon:long_name = "Longitude" ;
lon:standard_name = "longitude" ;
lon:short_name = "lon" ;
lon:axis = "X" ;
lon:units = "degrees_east" ;
float temp(time) ;
temp:long_name = "Temperature" ;
temp:standard_name = "sea_water_temperature" ;
temp:short_name = "temp" ;
temp:units = "Celsius" ;
temp:coordinates = "time lat lon alt" ;
temp:valid_range = 10., 35. ;
temp:_FillValue = -999.f ;
temp:observation_type = "measured" ;
答案 0 :(得分:3)
<强>更新强>: 下面的解决方案可以正常工作,但不:从中提取数据失败,正如John M.发现的那样(参见其他答案) )。我们认为我们已经发现维持单个维度是解决方案,但从四个维度到一个维度最终会导致错误。正如Sean A.所指出的,你无法使用NcML改变变量的形状。
原创“解决方案”(实际上并不奏效):
如果您的目标是使您的数据符合CF-1.6,则可以将该维度设为station
,其值为1。所以你可以这样做:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/usgs/data/file1.nc">
<remove type="dimension" name="lon"/>
<remove type="dimension" name="lat"/>
<remove type="dimension" name="z"/>
<dimension name="station" length="1"/>
<dimension name="name_strlen" length="20" />
<variable name="lat" shape="station"/>
<variable name="lon" shape="station"/>
<variable name="z" shape="station"/>
<variable name="temp" shape="time station"/>
<variable name="site" shape="station name_strlen" type="char">
<attribute name="standard_name" value="station_id" />
<attribute name="cf_role" value="timeseries_id" />
<values> my_station_001 </values>
</variable>
<attribute name="Conventions" value="CF-1.6" />
<attribute name="featureType" value="timeSeries" />
</netcdf>
答案 1 :(得分:3)
Rich的解决方案将针对这个非常具体的案例进行处理,但出于错误的原因。在NcML中,您可以删除维对象,但不能重新整形数据变量。对于这种特殊情况,当您尝试删除单个维度(大小为1)时,事情似乎会解决,因为它并没有真正改变数据在磁盘上的布局方式。例如,如果您使用Unidata的toolsUI使用Rich的答案中的NcML为temp变量执行ncdump,您将看到单例维度仍然存在且尚未真正删除。我不确定这会如何影响您的文件读取 - 我认为这取决于客户端。但是,如果您尝试删除非singelton维度,那么这将失败。
如果您确实想要正确地重塑数据,则必须重写netCDF文件。不幸的是,我所知道的并没有任何“捷径”。例如,如果在Unidata的toolsUI中使用Rich的NcML并尝试基于它写出新文件,则会出现错误,例如“ERROR:section(1)中的范围数量必须为= 0” Z“。这是因为netCDF文件中仍然存在单例维度,但是NcML文件试图强制范围为0.但是,如果你知道python,那么写一个脚本来重写你的netCDF文件应该非常简单。
请注意,使用NcML重塑变量的功能是我们定期听到的功能请求 - 将功能请求发送到support-netcdf-java@unidata.ucar.edu并不会有什么坏处。另请注意,Unidata是一个社区驱动的组织,Rich将在我们的用户委员会上,该委员会将在下个月召开会议。我建议他也在会议上提到功能请求。
干杯!
肖恩
答案 2 :(得分:2)
Rich是正确的,这样做的目的是尝试将我们的数据提升到CF-1.6,这样做的目的是让我们通过SOS提供数据。更具体地说,我们想使用ncSOS(基于TDS构建),这种特殊的SOS风格需要CF-1.6。在这方面,通过NCML的修改似乎有效(加上一些额外的修改;见下文)。
我不想修改数据集,其中一些可以追溯到几年前。 Sean关于客户端工具的观点也很重要,因为我们的许多用例都涉及需要变量维度为lat / lon的工具。因此,我们的解决方案是通过两个NCML'包装器'通过TDS提供一个数据集,一个用于ncSOS,另一个用于通过OPeNDAP访问需要lat / lon的特定客户端。
除了上面Rich的建议,为了在ncSOS中工作,我们不得不:
结果如下:
<attribute name="featureType" value="timeSeries" />
<remove type="dimension" name="lon"/>
<remove type="dimension" name="lat"/>
<remove type="dimension" name="z"/>
<dimension name="name_strlen" length="4"/>
<variable name="lat" shape=""/>
<variable name="lon" shape=""/>
<variable name="z" shape=""/>
<variable name="station_name" shape="name_strlen" type="char">
<attribute name="long_name" value="NS01" />
<attribute name="cf_role" value="timeseries_id" />
<values>NS01</values>
</variable>
<variable name="temp" shape="time">
<attribute name="coordinates" value="time lat lon z" />
</variable>
答案 3 :(得分:2)
为了跟进Jim上面的帖子,虽然Rich的NcML解决方案似乎最初起作用,但尝试通过OPeNDAP或ncSOS获取数据是不成功的,证实了Sean对此的怀疑。
目录显示成功,OPeNDAP表单显示CF-1.6的新维度和重新整形变量。此外,ncSOS GetCapabilities文档也会成功显示。
但是,尝试使用OPeNDAP表单下载某些数据存在问题。我无法在OPeNDAP表单上获得变量的子集。例如:
http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.html
如果我尝试使用此URL获取第一个临时值:
http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.ascii?temp [0:1:0]
它给了我这个错误:
Error {
code = 500;
message = "NcSDArray InvalidRangeException=Number of ranges in section (1) must be = 4";
};
唯一成功的是获得所有价值观:
http://oos.soest.hawaii.edu/thredds-test/dodsC/hioos/nss/ns01/ns01_2012_02_23.nc.ascii?temp [0:1:359]
此外,尝试通过ncSOS GetObservation获取数据也会失败。尝试了以下URL:
这导致threddsServlet.log中出现以下错误消息:
2013-10-02T09:03:44.844 -1000 [1288472818][ 2602] INFO - threddsServlet - Remote host: 128.171.151.240 - Request: "GET /thredds-test/sos/hioos/nss/ns01agg.ncml?service=SOS&version=1.0.0&request=GetObs
ervation&responseFormat=text%2Fxml%3Bsubtype%3D%22om%2F1.0.0%22&offering=NS01&observedProperty=temp&procedure=urn:ioos:station:org.pacioos:NS01 HTTP/1.1"
2013-10-02T09:03:44.845 -1000 [1288472819][ 2602] INFO - com.asascience.ncsos.controller.SosController - Handling SOS metadata request.
2013-10-02T09:03:45.614 -1000 [1288473588][ 2602] ERROR - ucar.nc2.Structure - Structure.IteratorRank1.readNext()
ucar.ma2.InvalidRangeException: Number of ranges in section (1) must be = 4
at ucar.ma2.Section.fill(Section.java:144)
at ucar.nc2.Variable.read(Variable.java:673)
at ucar.nc2.Variable.read(Variable.java:647)
at ucar.nc2.ncml.AggregationOuterDimension$DatasetOuterDimension.read(AggregationOuterDimension.java:774)
at ucar.nc2.ncml.AggregationOuterDimension.reallyRead(AggregationOuterDimension.java:293)
at ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)
at ucar.nc2.Variable.read(Variable.java:673)
at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:553)
at ucar.nc2.dataset.VariableDS._read(VariableDS.java:533)
at ucar.nc2.Variable.read(Variable.java:673)
at ucar.nc2.Variable.read(Variable.java:647)
at ucar.nc2.dataset.StructurePseudoDS.reallyRead(StructurePseudoDS.java:193)
at ucar.nc2.Variable._read(Variable.java:861)
at ucar.nc2.Variable.read(Variable.java:673)
at ucar.nc2.Variable.read(Variable.java:619)
at ucar.nc2.Structure.readStructure(Structure.java:378)
at ucar.nc2.Structure$IteratorRank1.readNext(Structure.java:464)
at ucar.nc2.Structure$IteratorRank1.next(Structure.java:447)
at ucar.nc2.ft.point.PointIteratorFromStructureData.nextStructureData(PointIteratorFromStructureData.java:103)
at ucar.nc2.ft.point.PointIteratorFromStructureData.hasNext(PointIteratorFromStructureData.java:68)
at ucar.nc2.ft.point.PointCollectionImpl.calcBounds(PointCollectionImpl.java:128)
at com.asascience.ncsos.util.DatasetHandlerAdapter.calcBounds(DatasetHandlerAdapter.java:122)
at com.asascience.ncsos.cdmclasses.TimeSeries.setData(TimeSeries.java:243)
at com.asascience.ncsos.getobs.SOSGetObservationRequestHandler.setCDMDatasetForStations(SOSGetObservationRequestHandler.java:193)
at com.asascience.ncsos.getobs.SOSGetObservationRequestHandler.<init>(SOSGetObservationRequestHandler.java:138)
at com.asascience.ncsos.service.SOSParser.enhanceGETRequest(SOSParser.java:197)
at com.asascience.ncsos.controller.SosController.handleSOSRequest(SosController.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:102)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2013-10-02T09:03:45.616 -1000 [1288473590][ 2602] ERROR - com.asascience.ncsos.util.DatasetHandlerAdapter - Could not calculate the bounds of the PointFeatureCollection NS01
Structure.Iterator.readNext()
2013-10-02T09:03:45.616 -1000 [1288473590][ 2602] ERROR - com.asascience.ncsos.cdmclasses.baseCDMClass - TimeSeries - setData; exception:
java.lang.NullPointerException
2013-10-02T09:03:45.616 -1000 [1288473590][ 2602] ERROR - com.asascience.ncsos.service.SOSParser - java.lang.NullPointerException
2013-10-02T09:03:45.617 -1000 [1288473591][ 2602] ERROR - com.asascience.ncsos.controller.SosController -
2013-10-02T09:03:45.817 -1000 [1288473791][ 2602] INFO - threddsServlet - Request Completed - 200 - -1 - 973:1
答案 4 :(得分:2)
NcML now(自4.4版开始)具有删除长度为1的维度的操作,例如:
<variable name="temp">
<logicalReduce dimNames="lat lon" />
</variable>
见