我正在尝试使用Flume和Hive进行推特分析。为了从twitter获取推文,我已经在flume.conf文件中设置了所有必需的参数(consumerKey,consumerSecret,accessToken和accessTokenSecret)。
TwitterAgent.sources = Twitter TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS TwitterAgent.sources.Twitter.type =
com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = <consumerKey>
TwitterAgent.sources.Twitter.consumerSecret = <consumerSecret>
TwitterAgent.sources.Twitter.accessToken = <accessToken>
TwitterAgent.sources.Twitter.accessTokenSecret = <accessTokenSecret
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics,
bigdata, cloudera, data science, data scientiest, business
intelligence, mapreduce, data warehouse, data warehousing, mahout,
hbase, nosql, newsql, businessintelligence, cloudcomputing
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs TwitterAgent.sinks.HDFS.hdfs.path
= hdfs://localhost:9000/user/flume/tweets/ TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100
我已经使用bash rc为flume tar ball和flume source snapshot jar文件设置了类路径。
export FLUME_HOME=/home/students/apache-flume-1.4.0-bin
export FLUME_SRC=/home/students/flume-sources-1.0-SNAPSHOT.jar
export PATH=$FLUME_HOME/bin:$FLUME_SRC/bin:$PATH
当我运行水槽剂时
flume-ng agent --conf-file twitter_flume.conf --name TwitterAgent -Dflume.root.logger=INFO,console -n TwitterAgent
我可以看到下面的日志跟踪,没有任何反应
15/06/23 23:41:55 INFO source.DefaultSourceFactory: Creating instance
of source Twitter, type com.cloudera.flume.source.TwitterSource
15/06/23 23:41:55 ERROR
node.PollingPropertiesFileConfigurationProvider: Failed to load
configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type:
com.cloudera.flume.source.TwitterSource, class:
com.cloudera.flume.source.TwitterSource at
org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
at
org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
at
org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
at
org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at
org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744) Caused by:
java.lang.ClassNotFoundException:
com.cloudera.flume.source.TwitterSource at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:190) at
org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)
... 11 more
我知道为什么当我设置了flume source.jar时会抛出这个错误。请帮我解决这个问题。
答案 0 :(得分:1)
您没有设置类路径,而是设置PATH(用于查找可执行二进制文件,而不是Java .jar文件)。
您可以在Flume conf目录的flume-env.sh文件中设置FLUME_CLASSPATH变量;或在命令行上添加-classpath <path/to/the/jar>
选项。
答案 1 :(得分:1)
1。这是文件 /usr/lib/flume-ng/conf/flume.conf :
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type= com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.keywords = Hadoop,BigData
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://quickstart.cloudera:8020/user/cloudera/flume/tweets/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100
2. 将以下 flume-env.sh.template 文件重命名为 flume-env.sh
〜] $ sudo cp /usr/lib/flume-ng/conf/flume-env.sh.template /usr/lib/flume-ng/conf/flume-env.sh
3. 在 flume-env.sh 文件中设置 JAVA_HOME 和 FLUME_CLASSPATH :
export JAVA_HOME = / usr / java / jdk1.7.0_67-cloudera
FLUME_CLASSPATH = “/ usr / lib中/水槽-纳克/ LIB /水槽来源-1.0-SNAPSHOT.jar”
4. 如果您的系统上没有找到“/ usr / lib / flume-ng / lib / flume-sources-1.0-SNAPSHOT.jar”然后从谷歌下载 apache-flume-1.6.0-bin 并将其 lib 文件夹复制到当前的 lib 文件夹。
确保在lib文件夹中提供flume-sources-1.0-SNAPSHOT.jar文件。
4.1。重命名旧的lib文件夹
4.2。下载并放置在cloudera桌面上并执行以下操作:
〜] $ sudo mv / usr / lib / flume-ng / lib / usr / lib / flume-ng / lib_cloudera
〜] $ sudo mv /home/cloudera/Desktop/apache-flume-1.6.0-bin/lib / usr / lib / flume-ng / lib
5. 现在运行Flume Agent Command:
〜] $ flume-ng agent --conf-file /usr/lib/flume-ng/conf/flume.conf --name TwitterAgent -Dflume.root.logger = INFO,console -n TwitterAgent
这应该成功运行。 一切都好。
答案 2 :(得分:0)
我认为ServiceTracker
不再工作了。试试com.cloudera.flume.source.TwitterSource
答案 3 :(得分:0)
抱歉,它确实有效,但要确保你的水槽/ lib中有所有的罐子。按照以下所有步骤操作:http://bigdatanalysis.blogspot.com.es/2014/02/collecting-tweets-in-hadoop-using-flume.html