hadoop + Flume + HDFS IO错误+ ConnectException

时间:2015-06-04 02:35:00

标签: hadoop hdfs flume-ng

我正在使用Cloudera Manager CDH 5.4.2,也安装了Flume,我无法保存从Twitter获得的信息

当我运行水槽代理时,它开始运行正常但在尝试将新事件数据写入hdfs时最终会出错。  我收到以下错误:

 INFO org.apache.flume.sink.hdfs.BucketWriter: Creating hdfs://192.168.109.6:8020/user/flume/tweets/2015/06/03/06//FlumeData.1433311217583.tmp

WARN org.apache.flume.sink.hdfs.HDFSEventSink: HDFS IO error
java.net.ConnectException: Call From cluster-05.xxxx.com/192.168.109.6 to cluster-05.xxxx.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

我所做的配置是:

flume-conf.property:

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://192.168.109.6:8020/user/flume/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

我使用了以下插件:

flume-sources-1.0-SNAPSHOT.jar
twitter4j-core-2.2.6.jar
twitter4j-media-support-2.2.6.jar
twitter4j-stream-2.2.6.jar

(我为twitter4j替换了twitter4j- -3.0.3.jar的版本 - -2.2.6.jar)

也是使用hdfs用户的目录

hadoop fs -ls /user/flume : 
drwxrwxrwx - flume flume  /user/flume/tweets

core-site.xml(at / hadoop / conf)我添加了:

<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8020</value>
</property>

我还在我离开Flume Agent作为HDFS用户的主机上运行hadoop dfsadmin -safemode

在此问题上,我非常感谢您的帮助。

0 个答案:

没有答案