Question

我正在尝试将protobuf用于logstash编解码器。我跟着：

但是，我无法启动我的logstash。

bin/logstash -f logstash-test-1.conf --config.reload.automatic --debug

这是错误：

...
[2017-09-11T18:46:44,887][DEBUG][logstash.inputs.twitter  ] config LogStash::Inputs::Twitter/@rate_limit_reset_in = 300
[2017-09-11T18:46:44,970][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"protobuf", :type=>"codec", :class=>LogStash::Codecs::Protobuf}
[2017-09-11T18:46:44,974][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@class_name = "Twitter"
[2017-09-11T18:46:44,974][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@include_path = ["/Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb"]
[2017-09-11T18:46:44,974][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@id = "16ab258d8e30f65d80c0bb85353e47740f7c9971-2"
[2017-09-11T18:46:44,975][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@enable_metric = true
[2017-09-11T18:46:44,975][DEBUG][logstash.codecs.protobuf ] Including protobuf file: /Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb
[2017-09-11T18:46:44,982][ERROR][logstash.agent           ] Cannot create pipeline {
:reason=>"Twitter is not a class", 
:backtrace=>[
"/Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb:9:in `(root)'", 
"org/jruby/RubyKernel.java:1040:in `require'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:1:in `(root)'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:163:in `require_with_metadata_analysis'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:207:in `require_pb_path'", 
"org/jruby/RubyArray.java:1613:in `each'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:71:in `register'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:71:in `register'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/codecs/base.rb:20:in `initialize'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/pipeline.rb:109:in `plugin'", 
"org/jruby/RubyKernel.java:1079:in `eval'", 
"(eval):12:in `initialize'", "/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/pipeline.rb:72:in `initialize'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/pipeline.rb:156:in `initialize'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/runner.rb:314:in `execute'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/runner.rb:209:in `run'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-09-11T18:46:44,988][DEBUG][logstash.agent           ] starting agent
[2017-09-11T18:46:44,991][DEBUG][logstash.agent           ] Starting puma
[2017-09-11T18:46:44,991][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-09-11T18:46:44,991][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2017-09-11T18:46:44,993][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-09-11T18:46:44,994][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-09-11T18:46:44,994][DEBUG][logstash.api.service     ] [api-service] start

为什么我收到错误：Twitter is not a class？

这是我的logstash配置：

...
output {
  kafka {
    bootstrap_servers => "localhost:9092"
    codec => protobuf {
      class_name => "Twitter"
      include_path => ['/Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb']
    }
    topic_id => "blockchain-tweets"
  }
#  stdout {
#    codec => rubydebug
#  }
}

当我使用Kafka作为logstash的输出时，这只会出错。当我使用Kafka作为输入并添加protobuf的编解码器时，它可以成功加载pb.rb类。

这是我的TwitterSimple.proto：

syntax = "proto3";

message Twitter {
  string message = 1;
}

以下是TwitterSimple.pb.rb后生成的ruby-protoc TwitterSimple.proto：

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

# forward declarations
class Twitter < ::ProtocolBuffers::Message; end

class Twitter < ::ProtocolBuffers::Message
  set_fully_qualified_name "Twitter"

  optional :string, :message, 1
end

Answer 1

我认为最近才增加了对PB 3的支持您可能需要升级到该版本并进行适当的配置更改。

针对logstash编解码器

1 个答案: