我已安装(来自源代码)cygnus 0.7.1,配置完成后(MySQL和HDFS接收器)我可以毫无问题地启动它。当我将cygnus订阅到orion上下文时,它会收到信息,但是MySQL和HDFS存在问题。这是日志:
15/03/17 13:58:52 INFO handlers.OrionRestHandler: Starting transaction (1426597123-891-0000000000)
15/03/17 13:58:52 INFO handlers.OrionRestHandler: Received data ({ "subscriptionId" : "5508250c1860a36e55240c84", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "ubk-temp", "isPattern" : "false", "id" : "ubk:temp:1", "attributes" : [ { "name" : "temperature", "type" : "float", "value" : "11" } ] }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
15/03/17 13:58:52 INFO handlers.OrionRestHandler: Event put in the channel (id=1549700267, ttl=10)
15/03/17 13:58:52 INFO sinks.OrionSink: Event got from the channel (id=1549700267, headers={fiware-servicepath=ubktemp, destination=ubk_temp_1_ubk-temp, content-type=application/json, fiware-service=ubikwa, ttl=10, transactionId=1426597123-891-0000000000, timestamp=1426597132511}, bodyLength=462)
15/03/17 13:58:52 INFO sinks.OrionSink: Event got from the channel (id=1549700267, headers={fiware-servicepath=ubktemp, destination=ubk_temp_1_ubk-temp, content-type=application/json, fiware-service=ubikwa, ttl=10, transactionId=1426597123-891-0000000000, timestamp=1426597132511}, bodyLength=462)
15/03/17 13:58:52 INFO sinks.OrionMySQLSink: [mysql-sink] Persisting data at OrionMySQLSink. Database: ubikwa, Table: ubktemp_ubk_temp_1_ubk-temp, Timestamp: 2015-03-17T13:58:52.511, Data (attrs): {temperature=11}, (metadata): {temperature_md=[]}
15/03/17 13:58:53 INFO sinks.OrionHDFSSink: [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (ubikwa/ubktemp/ubk_temp_1_ubk-temp/ubk_temp_1_ubk-temp.txt), Data ({"recvTime":"2015-03-17T13:58:52.511","temperature":"11", "temperature_md":[]})
15/03/17 13:58:53 WARN sinks.OrionSink: Bad context data (Table 'ubikwa.ubktemp_ubk_temp_1_ubk-temp' doesn't exist)
15/03/17 13:58:53 INFO sinks.OrionSink: Finishing transaction (1426597123-891-0000000000)
MySQL接收器不会引发任何错误,但不会创建表。并且HDFS接收器似乎无法创建文件。我以前安装了cygnus 0.6,它使用相同的配置。
修改
这是我的配置:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink
cygnusagent.channels = hdfs-channel mysql-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = ubikwa
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = ubktemp
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts de
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.de.matching_table = /opt/cygnus/conf/matching_table.conf
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# Comma-separated list of FQDN/IP address regarding the Cosmos Namenode endpoints
# If you are using Kerberos authentication, then the usage of FQDNs instead of IP addresses is mandatory
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = ***
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = ***
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# Hive FQDN/IP address of the Hive server
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
# Kerberos-based authentication enabling
cygnusagent.sinks.hdfs-sink.krb5_auth = false
# Kerberos username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
# Kerberos password
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
# Kerberos login file
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
# Kerberos configuration file
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf
# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = ***
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = ***
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
任何提示?
由于
答案 0 :(得分:2)
我相信它是因为您在OrionMySQLSink的配置中使用参数列
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column
在文档中声明,当使用列数据库时,必须在启动cygnus之前创建表。当您使用行时,将在首次插入之前自动创建所有8行。
在表格中,我们可以找到两个选项:
像往常一样修复了8个字段的行:recvTimeTs,recvTime,entityId,entityType,attrName,attrType,attrValue和attrMd。这些表格 如果表没有,则在执行时创建(和数据库) 先前存在于行插入中。关于attrValue,在其中 最简单的形式,这个值只是一个字符串,但自从Orion 0.11.0起 可以是Json对象或Json数组。关于attrMd,它包含一个 Json中属性的元数据数组的字符串序列化 (如果属性没有元数据,则插入空数组[]), 每个实体的属性有两列(一个用于值,另一个用于元数据),另外还有一个关于接收的列 数据的时间(recv_time)。这种表(和数据库) 必须先提供给天鹅座的执行,因为 每个实体可能具有不同数量的属性,并且 通知必须确保通知每个属性的值。
连接器关于内部表示的行为 数据的配置由配置参数控制, attr_persistence,其值可以是行或列。
答案 1 :(得分:2)
正如Petark所说,“列模式”不会自动创建表格,而且必须由您提供。为什么?原因是,根据您向Orion CB发出的订阅,以便向Cygnus发送通知,此类通知可能包括一些次要的更新,有时会设置其他非常不同的属性等。
例如,让我们考虑一个名为“car”的实体,其中包括“速度”,“位置”和“油位”。然后你可以说“每次速度变化时都会通知天鹅座,但只发送速度值。但同时你可以说”每次油位变化时都会通知天鹅座,并发送所有属性的值。“如果汽车开始移动,只有速度和位置改变,但不是油位,然后只有速度更新将通知天鹅座,天鹅座无法随时了解其他属性。
因此,如果您想要拥有所有3个属性的数据行,那么您必须自己配置表。顺便说一句,拥有这样的订阅示例将导致表中存在大量“speed-value,null,null”行。
与“行模式”的区别在于,与通知属性的数量无关,将为每个通知的属性添加新行,使所有行具有相同的格式(entityId,entitytType,attrName,attrType,attrValue ,attrMd);这些格式可以由Cygnus自动配置。