Question

我想通过logstash-output-webhdfs向hdfs发送数据。 logstash的配置是：

input{
    file{
            path => "/root/20160315.txt"
    }
}

output{
    webhdfs{
            host => "x.x.x.x"
            path => "/user/logstash/dt=%{+YYYY-MM-dd}/logstash-%{+HH}.log"
            user => "logstash"
    }
    stdout{
            codec => rubydebug
    }
}

但是我收到以下错误：

Failed to flush outgoing items {:outgoing_count=>1, :exception=>"WebHDFS::ServerError", :backtrace=>["/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:351:in `request'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:270:in `operate_requests'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:73:in `create'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-2.0.4/lib/logstash/outputs/webhdfs.rb:211:in `write_data'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-2.0.4/lib/logstash/outputs/webhdfs.rb:195:in `flush'", "org/jruby/RubyHash.java:1342:in `each'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-2.0.4/lib/logstash/outputs/webhdfs.rb:183:in `flush'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-2.0.4/lib/logstash/outputs/webhdfs.rb:166:in `receive'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/pipeline.rb:305:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/pipeline.rb:305:in `output_batch'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/pipeline.rb:236:in `worker_loop'", "/root/logstash-2.3.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.0-java/lib/logstash/pipeline.rb:205:in `start_workers'"], :level=>:warn}

有没有人遇到过这个问题。

Answer 1

似乎你应该将logstash-output-webhdfs的用户选项设置为hdfs supergroup用户，这是用来启动hdf的用户。例如，如果你使用root来运行start-dfs.sh bash，那么用户选项shuold是root。

此外，您应该编辑/ etc / hosts，添加hdfs集群节点路由列表。

Answer 2

确保您的 Hadoop 集群已启用 WebHdfs。为此，请在您的名称节点和数据节点中启用 WebHdfs。

在 hdfs-site.xml 中

<property>
  <name>hfs.webhdfs.enabled</name>
  <value>true</value>
</property>

logstash-output-webhdfs无法刷新传出项目

2 个答案: