FIWARE Cygnus - ＆gt; cartodb sinks.NGSISink：持久性错误，400错误请求

Question

我正在尝试将天鹅座（1.4.0_SNAPSHOT）连接到cartodb。我在本地运行它，我使用脚本向天鹅座发送通知。该脚本运行正常，但天鹅座说：

ERROR sinks.NGSISink: Persistence error (The query 'INSERT INTO jcarneroatos.x002fpeoplelocation (recvtime,fiwareservicepath,entityid,entitytype,the_geom) VALUES ('2016-10-31T19:04:00.994Z','/peoplelocation','Person:1','Person',ST_SetSRID(ST_MakePoint({"coordinates":[-4.423032856,36.721290055]), 4326))' could not be executed. CartoDB response: 400 Bad Request)

任何人都知道会发生什么？下面我把我的配置文件作为信息，谢谢！

我在CARTO的用户名是“jcarneroatos”，域名为https://jcarneroatos.carto.com。这是我用来模拟Orion Context Broker的通知的脚本：

#/bin/bash
HOST=localhost
PORT=5050
SERVICE=jcarneroatos
SUBSERVICE=/peoplelocation

#send notification
NOTIFICATION=$(\
curl http://$HOST:$PORT/notify \
    -v -s -S \
    --header "Content-Type: application/json; charset=utf-8" \
    --header 'Accept: application/json' \
    --header "Fiware-Service: $SERVICE" \
    --header "Fiware-ServicePath: $SUBSERVICE" \
    -d '
    {
        "contextResponses": [
            {
                "contextElement": {
                    "attributes": [
                        {
                            "metadatas": [
                                {
                                    "name": "location",
                                    "type": "string",
                                    "value": "WGS84"
                                }
                            ],
                            "name": "location",
                            "type": "geo:json",
                            "value": {
                                "coordinates": [
                                    -4.423032856,
                                    36.721290055
                                ],
                                "type": "Point"
                            }
                        }
                    ],
                    "id": "Person:1",
                    "isPattern": "false",
                    "type": "Person"
                },
                "statusCode": {
                    "code": "200",
                    "reasonPhrase": "OK"
                }
            }
        ],
        "originator": "localhost",
        "subscriptionId": "58178396634ded66caac35b2"
    }')
if [ -z "$NOTIFICATION" ]; then
    echo "Ok"
else
    echo $NOTIFICATION
fi

这是cartodb数据集的结构：

x002fpeoplelocation
cartodb_id | the_geom | entityid | entitytype | fiwareservicepath | recvtime
  number   | geometry |  string  |   string   |      string       |   date

这是天鹅座配置文件：

cygnusagent.sources = http-source
cygnusagent.sinks = cartodb-sink
cygnusagent.channels =cartodb-channel

cygnusagent.sources.http-source.channels = cartodb-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = jcarneroatos
cygnusagent.sources.http-source.handler.default_service_path = /peoplelocation
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /home/cygnus/APACHE_FLUME_HOME/conf/grouping_rules.conf

cygnusagent.sinks.cartodb-sink.type = com.telefonica.iot.cygnus.sinks.NGSICartoDBSink
cygnusagent.sinks.cartodb-sink.channel = cartodb-channel
cygnusagent.sinks.cartodb-sink.enable_grouping = false
cygnusagent.sinks.cartodb-sink.enable_name_mappings = false
cygnusagent.sinks.cartodb-sink.enable_lowercase = false
cygnusagent.sinks.cartodb-sink.data_model = dm-by-service-path
cygnusagent.sinks.cartodb-sink.keys_conf_file = /home/cygnus/APACHE_FLUME_HOME/conf/cartodb_keys.conf
cygnusagent.sinks.cartodb-sink.flip_coordinates = false
cygnusagent.sinks.cartodb-sink.enable_raw = true
cygnusagent.sinks.cartodb-sink.enable_distance = false
cygnusagent.sinks.cartodb-sink.batch_size = 100
cygnusagent.sinks.cartodb-sink.batch_timeout = 30
cygnusagent.sinks.cartodb-sink.batch_ttl = 10
cygnusagent.sinks.cartodb-sink.backend.max_conns = 500
cygnusagent.sinks.cartodb-sink.backend.max_conns_per_route = 100

cygnusagent.channels.cartodb-channel.type = memory
cygnusagent.channels.cartodb-channel.capacity = 1000
cygnusagent.channels.cartodb-channel.transactionCapacity = 100

最后是cartodb_keys.conf文件（没有密钥）：

{
   "cartodb_keys": [
      {
         "username": "jcarneroatos",
         "endpoint": "https://jcarneroatos.carto.com",
         "key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"   
      }
   ]
}

更新 在DEBUG模式下执行Cygnus并检查日志后，似乎CARTO正在返回：

{"error":["syntax error at or near \"{\""]}

这是完整的日志：http://pastebin.com/p9VyUU8n

Answer 1

问题是geo:json目前不支持NGSICartoDBSink类型。根据Orion Context Broker规范，这个接收器理解了通知地理定位属性的方法。这些：

• 使用geo:point类型，并使用格式"latitude, longitude"在值字段中发送坐标。
• 使用类型为location且值为string的{​​{1}}元数据，并使用格式WGS84在值字段中发送坐标。

请注意：

• 上述选项是特殊的，即不能同时使用。
• 在Orion中不推荐使用"latitude, longitude"元数据，但仍然可以使用它。

虽然支持location（我将开始处理，可以在此sprint /月内准备就绪），我建议您使用geo:json类型。

编辑1

我在这里添加了一个Cygnus执行示例，当收到涉及地理定位属性（geo:point类型）的通知时。

Cygnus版本：

geo:point

天鹅座配置：

1.6.0

创建表格：

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = raw-sink
cygnus-ngsi.channels = raw-channel

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = raw-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp

cygnus-ngsi.sinks.raw-sink.type = com.telefonica.iot.cygnus.sinks.NGSICartoDBSink
cygnus-ngsi.sinks.raw-sink.channel = raw-channel
cygnus-ngsi.sinks.raw-sink.enable_grouping = false
cygnus-ngsi.sinks.raw-sink.keys_conf_file = /usr/cygnus/conf/cartodb_keys.conf
cygnus-ngsi.sinks.raw-sink.swap_coordinates = false
cygnus-ngsi.sinks.raw-sink.enable_raw = true
cygnus-ngsi.sinks.raw-sink.enable_distance = false
cygnus-ngsi.sinks.raw-sink.enable_raw_snapshot = false
cygnus-ngsi.sinks.raw-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.raw-sink.batch_size = 50
cygnus-ngsi.sinks.raw-sink.batch_timeout = 10
cygnus-ngsi.sinks.raw-sink.batch_ttl = 0
cygnus-ngsi.sinks.raw-sink.batch_retries = 5000

cygnus-ngsi.channels.raw-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.raw-channel.capacity = 1000
cygnus-ngsi.channels.raw-channel.transactionCapacity = 100

模拟通知的脚本：

$ curl -X GET -G "https://<my_user>.cartodb.com/api/v2/sql?api_key=<my_key>" --data-urlencode "q=CREATE TABLE x002ftestxffffx0043ar1xffffx0043ar (recvTime text, fiwareServicePath text, entityId text, entityType text, speed float, speed_md text, the_geom geometry(POINT,4326))"
{"rows":[],"time":0.005,"fields":{},"total_rows":0}

脚本执行：

$ cat notification.sh
#!/bin/sh

URL=$1

if [ "$2" != "" ]
then
   SERVICE=$2
else
   SERVICE=default
fi

if [ "$3" != "" ]
then
   SERVICE_PATH=$3
else
   SERVICE_PATH=/
fi

curl $URL -v -s -S --header 'Content-Type: application/json; charset=utf-8' --header 'Accept: application/json' --header 'User-Agent: orion/0.10.0' --header "Fiware-Service: $SERVICE" --header "Fiware-ServicePath: $SERVICE_PATH" -d @- 
<<EOF
{
  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",
  "originator" : "localhost",
  "contextResponses" : [
    {
      "contextElement" : {
        "attributes" : [
          {
            "name" : "speed",
            "type" : "float",
            "value" : "$6"
          },
          {
            "name" : "the_geom",
            "type" : "geo:point",
            "value" : "$4, $5"
          }
        ],
        "type" : "Car",
        "isPattern" : "false",
        "id" : "Car1"
      },
      "statusCode" : {
        "code" : "200",
        "reasonPhrase" : "OK"
      }
    }
  ]
}
EOF

Cygnus在收到通知时登录：

$ ./notification.sh http://localhost:5050/notify <my_user> /test 40.40 -3.4 120
*   Trying ::1...
* Connected to localhost (::1) port 5050 (#0)
> POST /notify HTTP/1.1
> Host: localhost:5050
> Content-Type: application/json; charset=utf-8
> Accept: application/json
> User-Agent: orion/0.10.0
> Fiware-Service: <my_user>
> Fiware-ServicePath: /test
> Content-Length: 569
> 
* upload completely sent off: 569 out of 569 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: Jetty(6.1.26)
< 
* Connection #0 to host localhost left intact

获取数据：

time=2016-12-02T13:48:27.310UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[282] : [NGSIRestHandler] Starting internal transaction (eb73b8d5-af9b-48ea-8ce7-ff21edc957f3)
time=2016-12-02T13:48:27.312UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[299] : [NGSIRestHandler] Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "speed",            "type" : "float",            "value" : "120"          },          {            "name" : "the_geom",            "type" : "geo:point",            "value" : "40.40, -3.4"          }        ],        "type" : "Car",        "isPattern" : "false",        "id" : "Car1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2016-12-02T13:48:36.404UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=persistRawAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[553] : [raw-sink] Persisting data at NGSICartoDBSink. Schema (<my_user>), Table (x002ftestxffffx0043ar1xffffx0043ar), Data (('2016-12-02T13:48:27.381Z','/test','Car1','Car',ST_SetSRID(ST_MakePoint(40.40,-3.4), 4326),'120','[]'))
time=2016-12-02T13:48:38.237UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv=<my_user> | subsrv=/test | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[514] : Finishing internal transaction (eb73b8d5-af9b-48ea-8ce7-ff21edc957f3)

编辑2

此答案仅在您拥有“企业”Carto帐户时有效。请看这个问题的其他答案。

Answer 2

最后，在与@Javi Carnero进行离线讨论后，我们发现Cygnus代码中存在一个与Carto区分“企业”和“个人”帐户的方式有关的错误。第一个允许每个用户在企业组织下使用PostgreSQL模式，而第二个模式具有名为public的“硬编码”模式。由于通知的FIWARE服务被用作模式名称，因此Cygnus无法正常使用“个人”帐户。

这样的错误已得到修复：

• 问题：https://github.com/telefonicaid/fiware-cygnus/issues/1382
• PR：https://github.com/telefonicaid/fiware-cygnus/issues/1393

在撰写本文时，错误已在master分支中修复。在sprint /月结束时，将发布Cygnus 1.7.0，包括此修复程序。

如果您拥有“企业”帐户，请注意我之前的回答是完全有效的。无论如何，我会编辑它以解释这个。