使用Apache Flume Error

时间:2017-11-11 07:01:21

标签: apache flume phoenix

我想使用Apache Flume将JSON数据传入Apache Phoenix,遵循在线指南http://kalyanbigdatatraining.blogspot.com/2016/10/how-to-stream-json-data-into-phoenix.html,但遇到以下错误。怎么解决?非常感谢!

我的环境列表为:

  • 的hadoop-2.7.3
  • hbase-1.3.1
  • 凤-4.12.0-HBase的-1.3滨
  • 水槽-1.7.0

在水槽中,我在$ FLUME_HOME / plugins.d / phoenix-sink / lib

中添加了与phoenix sink相关的罐子

公地-IO-2.4.jar
斜纹API-0.8.0.jar
斜纹发现-API-0.8.0.jar JSON-路径2.2.0.jar
斜纹共0.8.0.jar
斜纹发现核-0.8.0.jar 凤凰水槽-4.12.0-HBase的-1.3.jar
斜纹核心0.8.0.jar
斜纹动物园管理员-0.8.0.jar

2017-11-11 14:49:54,786(lifecycleSupervisor-1-1)[DEBUG - org.apache.phoenix.jdbc.PhoenixDriver $ 2.onRemoval(PhoenixDriver.java:159)] expiring localhost:2181:/ hbase因为EXPLICIT 2017-11-11 14:49:54,787(lifecycleSupervisor-1-1)[INFO - org.apache.hadoop.hbase.client.ConnectionManager $ HConnectionImplementation.closeZooKeeperWatcher(ConnectionManager.java:1712)]关闭zookeeper sessionid = 0x15fa8952cea00a6 2017-11-11 14:49:54,787(lifecycleSupervisor-1-1)[DEBUG - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:673)]结束会话:0x15fa8952cea00a6 2017-11-11 14:49:54,787(lifecycleSupervisor-1-1)[DEBUG - org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1306)]关闭会话客户端:0x15fa8952cea00a6 2017-11-11 14:49:54,789(lifecycleSupervisor-1-1-SendThread(localhost:2181))[DEBUG - org.apache.zookeeper.ClientCnxn $ SendThread.readResponse(ClientCnxn.java:818)]阅读回复sessionid: 0x15fa8952cea00a6,packet :: clientPath:null serverPath:null finished:false header :: 3,-11 replyHeader :: 3,2620,0 request :: null response :: null 2017-11-11 14:49:54,789(lifecycleSupervisor-1-1)[DEBUG - org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1290)]断开会话客户端:0x15fa8952cea00a6 2017-11-11 14:49:54,789(lifecycleSupervisor-1-1-SendThread(localhost:2181))[DEBUG - org.apache.zookeeper.ClientCnxn $ SendThread.run(ClientCnxn.java:1086)]抛出异常关闭会话0x15fa8952cea00a6的发送线程:无法从服务器sessionid 0x15fa8952cea00a6读取其他数据,可能服务器已关闭套接字 2017-11-11 14:49:54,789(lifecycleSupervisor-1-1)[INFO - org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:684)]会话:0x15fa8952cea00a6已关闭 2017-11-11 14:49:54,789(lifecycleSupervisor-1-1-EventThread)[INFO - org.apache.zookeeper.ClientCnxn $ EventThread.run(ClientCnxn.java:512)] EventThread关闭 2017-11-11 14:49:54,790(lifecycleSupervisor-1-1)[错误 - org.apache.flume.lifecycle.LifecycleSupervisor $ MonitorRunnable.run(LifecycleSupervisor.java:251)]无法启动SinkRunner:{policy:org .apache.flume.sink.DefaultSinkProcessor @ 2d2052a0 counterGroup:{name:null counters:{}}} - 异常如下。

  

java.lang.NoSuchMethodError:   org.apache.twill.zookeeper.ZKClientService.startAndWait()LCOM /谷歌/普通/ util的/并行/服务$国家;

at org.apache.phoenix.transaction.TephraTransactionContext.setTransactionClient(TephraTransactionContext.java:147)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.initTxServiceClient(ConnectionQueryServicesImpl.java:401)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:415)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$500(ConnectionQueryServicesImpl.java:257)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2384)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2360)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2360)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.flume.serializer.BaseEventSerializer.initialize(BaseEventSerializer.java:140)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:119)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:45)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

2017-11-11 14:49:54,792(lifecycleSupervisor-1-1)[INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:149)]组件类型:SINK,名称:Phoenix Sink__1停了下来 2017-11-11 14:49:54,792(lifecycleSupervisor-1-1)[INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:155)]关闭类型的公制:SINK,名称:Phoenix Sink__1 。 sink.start.time == 1510382993516

这是我的flume-agent.properties

agent.sources = exec
agent.channels = mem-channel
agent.sinks = phoenix-sink
agent.sources.exec.type = exec
agent.sources.exec.command = tail -F /Users/chenshuai1/tmp/users.json
agent.sources.exec.channels = mem-channel
agent.sinks.phoenix-sink.type = org.apache.phoenix.flume.sink.PhoenixSink
agent.sinks.phoenix-sink.batchSize = 10
agent.sinks.phoenix-sink.zookeeperQuorum = localhost
agent.sinks.phoenix-sink.table = users2
agent.sinks.phoenix-sink.ddl = CREATE TABLE IF NOT EXISTS users2 (userid BIGINT NOT NULL, username VARCHAR, password VARCHAR, email VARCHAR, country VARCHAR, state VARCHAR, city VARCHAR, dt VARCHAR NOT NULL CONSTRAINT PK PRIMARY KEY (userid, dt))
agent.sinks.phoenix-sink.serializer = json
agent.sinks.phoenix-sink.serializer.columnsMapping = {"userid":"userid", "username":"username", "password":"password", "email":"email", "country":"country", "state":"state", "city":"city", "dt":"dt"}
agent.sinks.phoenix-sink.serializer.partialSchema = true
agent.sinks.phoenix-sink.serializer.columns = userid,username,password,email,country,state,city,dt
agent.sinks.phoenix-sink.channel = mem-channel
agent.channels.mem-channel.type = memory
agent.channels.mem-channel.capacity = 1000
agent.channels.mem-channel.transactionCapacity = 100

0 个答案:

没有答案