我正在运行Flume从Twitter提取数据。执行此命令时,我收到此错误。我无法理解出了什么问题。
18/03/25 16:04:46 ERROR lifecycle.LifecycleSupervisor:无法启动 EventDrivenSourceRunner:{ 源:com.cloudera.flume.source.TwitterSource {名称:微博状态:空闲} - 以下是例外情况。显示java.lang.NullPointerException
我得到以下日志。
18/03/25 16:04:46 INFO node.AbstractConfigurationProvider: Channel MemCh connected to [Twitter, HDFS]
18/03/25 16:04:46 INFO node.Application: Starting new configuration:{ sourceRunners:{Twitter=EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6f044771 counterGroup:{ name:null counters:{} } }} channels:{MemCh=org.apache.flume.channel.MemoryChannel{name: MemCh}} }
18/03/25 16:04:46 INFO node.Application: Starting Channel MemCh
18/03/25 16:04:46 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: MemCh: Successfully registered new MBean.
18/03/25 16:04:46 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: MemCh started
18/03/25 16:04:46 INFO node.Application: Starting Sink HDFS
18/03/25 16:04:46 INFO node.Application: Starting Source Twitter
18/03/25 16:04:46 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: HDFS: Successfully registered new MBean.
18/03/25 16:04:46 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started
18/03/25 16:04:46 ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.NullPointerException
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:171)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
18/03/25 16:04:49 ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.NullPointerException
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:171)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
^C18/03/25 16:04:51 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 12
18/03/25 16:04:51 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider stopping
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS stopped
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.start.time == 1521993886561
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.stop.time == 1521993891665
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.batch.complete == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.batch.empty == 2
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.batch.underflow == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.connection.closed.count == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.connection.creation.count == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.connection.failed.count == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.event.drain.attempt == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.event.drain.sucess == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: MemCh stopped
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.start.time == 1521993886559
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.stop.time == 1521993891666
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.capacity == 10000
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.current.size == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.event.put.attempt == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.event.put.success == 0
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.event.take.attempt == 2
18/03/25 16:04:51 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemCh. channel.event.take.success == 0
以下是水槽配置:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemCh
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemCh
TwitterAgent.sources.Twitter.consumerKey=
TwitterAgent.sources.Twitter.consumerSecret=
TwitterAgent.sources.Twitter.accessToken=
TwitterAgent.sources.Twitter.accessTokenSecret=
TwitterAgent.sources.Twitter.keywords = google,microsoft,apple,facebook,tesla,oracle,infosys,tcs
TwitterAgent.sources.Twitter.language = en
TwitterAgent.sinks.HDFS.channel = MemCh
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/Twitter/PublicStream/day_key=%Y%m%d/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemCh.type = memory
TwitterAgent.channels.MemCh.capacity = 10000
TwitterAgent.channels.MemCh.transactionCapacity = 1000