配置cygnus.conf

时间:2015-05-07 12:28:54

标签: fiware fiware-cygnus

我需要使用mysql格式将数据存储在cosmos中,但我不能这样做。我已经检查过数据存储在cosmos中的文本文件中,但是当我输入hive时,没有任何表格显示我的名字。

以下是天鹅座的配置。这是对的吗?我的错误存储在mysql表中的哪个位置?

cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink
cygnusagent.channels = hdfs-channel mysql-channel

cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 10
cygnusagent.sources.http-source.interceptors = ts de
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
cygnusagent.sinks.hdfs-sink.cosmos_default_username = cristina.albaladejo
cygnusagent.sinks.hdfs-sink.cosmos_default_password = my_passw
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
cygnusagent.sinks.hdfs-sink.attr_persistence = column
cygnusagent.sinks.hdfs-sink.hive_host = 130.206.80.46
cygnusagent.sinks.hdfs-sink.hive_port = 10000
cygnusagent.sinks.hdfs-sink.krb5_auth = false
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf

# OrionMySQLSink configuration
cygnusagent.sinks.mysql-sink.channel = mysql-channel
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
cygnusagent.sinks.mysql-sink.mysql_host = 130.206.80.46
cygnusagent.sinks.mysql-sink.mysql_port = 3306
cygnusagent.sinks.mysql-sink.mysql_username = root
cygnusagent.sinks.mysql-sink.mysql_password = my_passw
cygnusagent.sinks.mysql-sink.attr_persistence = column

# hdfs-channel configuration
cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100

# mysql-channel configuration
cygnusagent.channels.mysql-channel.type = memory
cygnusagent.channels.mysql-channel.capacity = 1000
cygnusagent.channels.mysql-channel.transactionCapacity = 100

1 个答案:

答案 0 :(得分:0)

正如我在这个问题中的回答:

How to configure Cygnus to save in mysql

  

不会在列模式下自动创建数据库表。所以   你必须创建表格。

     

列看起来像:recvTime - datetime,field1,field2 ....   field1_md - varchar,field2_md - varchar ....

     

如果你改变了

     

cygnusagent.sinks.mysql-sink.attr_persistence = column

     

     

cygnusagent.sinks.mysql-sink.attr_persistence =创建行表   自动,但我更喜欢列方式来保存和处理数据。

您可以查看天鹅座日志,了解实际情况。但似乎是你必须使用字段声明

创建数据库和表

你也有我的样本配置,运行正常

您还可以使用ps aux | grep cygnus搜索cygnus proccess并使用kill XXXX(天鹅座进程数)将其删除

然后在像这样的终端中运行Cygnus

/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/YOURAGENTCONFIGFILE.conf -n cygnusagent -Dflume.root.logger=INFO,console

(您应该更改配置文件的名称以及运行它的cygnusagent)

我希望这会对你有所帮助。