Question

我在Ubuntu 14.04上使用Hadoop-1.2.1

我正在尝试使用Flume-1.6.0将数据从twitter传输到HDFS。我已经下载了flume-sources-1.0-SNAPSHOT.jar并将其包含在flume / lib文件夹中。我在conf / flume-env.sh中将flume-sources-1.0-SNAPSHOT.jar的路径设置为FLUME_CLASSPATH。这是我的flume agent conf文件：

＆＃13;

#setting properties of agent
Twitter-agent.sources=source1
Twitter-agent.channels=channel1
Twitter-agent.sinks=sink1

#configuring sources
Twitter-agent.sources.source1.type=com.cloudera.flume.source.TwitterSource
Twitter-agent.sources.source1.channels=channel1
Twitter-agent.sources.source1.consumerKey=<consumer-key>
Twitter-agent.sources.source1.consumerSecret=<consumer Secret>
Twitter-agent.sources.source1.accessToken=<access Toekn>
Twitter-agent.sources.source1.accessTokenSecret=<acess Token Secret>
Twitter-agent.sources.source1.keywords= morning, night, hadoop, bigdata

#configuring channels
Twitter-agent.channels.channel1.type=memory
Twitter-agent.channels.channel1.capacity=10000
Twitter-agent.channels.channel1.transactionCapacity=100

#configuring sinks
Twitter-agent.sinks.sink1.channel=channel1
Twitter-agent.sinks.sink1.type=hdfs
Twitter-agent.sinks.sink1.hdfs.path=flume/twitter/logs
Twitter-agent.sinks.sink1.rollSize=0
Twitter-agent.sinks.sink1.rollCount=1000
Twitter-agent.sinks.sink1.batchSize=100
Twitter-agent.sinks.sink1.fileType=DataStream
Twitter-agent.sinks.sink1.writeFormat=Text

＆＃13;

当我运行此代理时，我收到如下错误：

＆＃13;

15/06/22 14:14:49 INFO source.DefaultSourceFactory: Creating instance of source source1, type com.cloudera.flume.source.TwitterSource
15/06/22 14:14:49 ERROR node.PollingPropertiesFileConfigurationProvider: Unhandled error
java.lang.NoSuchMethodError: twitter4j.conf.Configuration.isStallWarningsEnabled()Z
	at twitter4j.TwitterStreamImpl.<init>(TwitterStreamImpl.java:60)
	at twitter4j.TwitterStreamFactory.<clinit>(TwitterStreamFactory.java:40)
	at com.cloudera.flume.source.TwitterSource.<init>(TwitterSource.java:64)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at java.lang.Class.newInstance(Class.java:442)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:44)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

＆＃13;

我的flume / lib文件夹已经有twitter4j-core-3.0.3.jar

如何纠正此错误？

Answer 1

我找到了解决这个问题的方法。由于flume-sources-1.0-SNAPSHOT.jar和twitter4j-stream-3.0.3.jar包含相同的FilterQuery.class，因此出现了jar冲突。所有twitter4j-3.xx都使用这个类，所以最好下载版本2.2.6（twitter4j-core，twitter4j-stream，twitter4j-media-support）的twitter jars并用这些新替换3.xx在flume / lib目录下下载了jar。

再次运行代理，Twitter数据将流式传输到HDFS。

Answer 2

更改 Twitter的agent.sources.source1.type = com.cloudera.flume.source.TwitterSource 同 TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource

使用flume将Twitter数据流式传输到Hadoop时出错

2 个答案: