如何使用LogStash在tomcat日志文件上创建索引

时间：2016-02-08 22:22:12

标签： tomcat elasticsearch logstash kibana-4 logstash-grok

我想从tomcat日志文件中进行一些日常分析，例如提出了多少错误和异常以及它们的类别等等。所以我选择ELK来做这个并且是日志索引的新手。

这是我的conf文件：

input {
    file {
        path => "\localhost.2016-01-09.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout {codec => "rubydebug"}
    elasticsearch {
        hosts => ["localhost:9200"]

    }
}

以下是日志文件的一些行：

09-Jan-2016 18:30:38.722 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Jan-2016 18:30:38.796 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\0-contact-statecollab-ws-15.12-SNAPSHOT-unknown-20151230-1152\WEB-INF\log4j.properties]
09-Jan-2016 18:30:38.901 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
09-Jan-2016 18:30:54.271 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Jan-2016 18:30:54.316 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\1-ohsms-ws-15.10.16-SNAPSHOT-unknown-20151119-1832\WEB-INF\log4j.properties]
09-Jan-2016 18:30:54.361 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
09-Jan-2016 18:31:14.627 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log ContextListener: contextInitialized()
09-Jan-2016 18:31:14.628 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log SessionListener: contextInitialized()
09-Jan-2016 18:43:18.329 INFO [http-apr-8080-exec-9] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Jan-2016 18:43:18.369 INFO [http-apr-8080-exec-9] org.apache.catalina.core.ApplicationContext.log Initializing log4j from [C:\tomcat\apache-tomcat-8.0.26\temp\5-exports-service-15.12-SNAPSHOT-unknown-20160108-1712\WEB-INF\log4j.properties]
09-Jan-2016 18:43:18.405 INFO [http-apr-8080-exec-9] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
09-Jan-2016 18:43:29.405 SEVERE [http-apr-8080-exec-9] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
 org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exports-ws': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: protected gov.hhs.acf.ohs.hses.model.exports.service.ExportsService gov.hhs.acf.ohs.hses.services.exports.ExportsWebService.exportsService; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [gov.hhs.acf.ohs.hses.model.exports.service.ExportsService] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true), @org.springframework.beans.factory.annotation.Qualifier(value=exportsService)}
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hses_service] is not bound in this Context. Unable to find [hses_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactService' defined in class path resource [contactContext.xml]: Cannot resolve reference to bean 'contactPgTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgTransactionManager' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'contactPgSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgSessionFactory' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hses_service] is not bound in this Context. Unable to find [hses_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drsExportsService' defined in class path resource [drsExportsContext.xml]: Cannot resolve reference to bean 'drsExportTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drsExportTransactionManager' defined in class path resource [drsExportsContext.xml]: Cannot resolve reference to bean 'drsExportSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drsExportSessionFactory' defined in class path resource [drsExportsContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hses_service] is not bound in this Context. Unable to find [hses_service].
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'exports-ws': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: protected gov.hhs.acf.ohs.hhhh.model.exports.service.ExportsService gov.hhs.acf.ohs.hhhh.services.exports.ExportsWebService.exportsService; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [gov.hhs.acf.ohs.hhhh.model.exports.service.ExportsService] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true), @org.springframework.beans.factory.annotation.Qualifier(value=exportsService)}
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hhhh_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactService' defined in class path resource [contactContext.xml]: Cannot resolve reference to bean 'contactPgTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgTransactionManager' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'contactPgSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'contactPgSessionFactory' defined in class path resource [contactDaoPgContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hhhh_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ccrRegistrantExportService' defined in class path resource [ccrRegistrantExportPgContext.xml]: Cannot resolve reference to bean 'ccrRegistrantExportTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ccrRegistrantExportTransactionManager' defined in class path resource [ccrRegistrantExportPgContext.xml]: Cannot resolve reference to bean 'ccrRegistrantExportSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ccrRegistrantExportSessionFactory' defined in class path resource [ccrRegistrantExportPgContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hhhh_service].
Related cause: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drsExportsService' defined in class path resource [drsExportsContext.xml]: Cannot resolve reference to bean 'drsExportTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drsExportTransactionManager' defined in class path resource [drsExportsContext.xml]: Cannot resolve reference to bean 'drsExportSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drsExportSessionFactory' defined in class path resource [drsExportsContext.xml]: Cannot resolve reference to bean 'gisDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gisDataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [hhhh_service] is not bound in this Context. Unable to find [hhhh_service].

当我运行脚本时：“logstash -f first-pipeline.conf”，它说

  

Logstash启动完成

但是当我使用以下行检查索引时：

  

http://localhost:9200/_cat/indices

没有创建这样的索引。我错过了什么吗？我不想更改服务器上的任何tomcat配置，并希望从tomcat日志文件而不是运行服务器获取每日​​报告

我切换到Ubuntu更多地玩它并检查弹性搜索的日志。它说：

[2016-02-10 12:15:32,496][INFO ][node                     ] [Archie Corrigan] version[2.2.0], pid[6365], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-02-10 12:15:32,496][INFO ][node                     ] [Archie Corrigan] initializing ...
[2016-02-10 12:15:33,232][INFO ][plugins                  ] [Archie Corrigan] modules [lang-expression, lang-groovy], plugins [], sites []
[2016-02-10 12:15:33,269][INFO ][env                      ] [Archie Corrigan] using [1] data paths, mounts [[/ (/dev/loop0)]], net usable_space [8.7gb], net total_space [16.9gb], spins? [possibly], types [ext4]
[2016-02-10 12:15:33,269][INFO ][env                      ] [Archie Corrigan] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-02-10 12:15:35,932][INFO ][node                     ] [Archie Corrigan] initialized
[2016-02-10 12:15:35,932][INFO ][node                     ] [Archie Corrigan] starting ...
[2016-02-10 12:15:36,044][INFO ][transport                ] [Archie Corrigan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2016-02-10 12:15:36,052][INFO ][discovery                ] [Archie Corrigan] elasticsearch/CNUHuSKySm6osx2heQAIqg
[2016-02-10 12:15:39,132][INFO ][cluster.service          ] [Archie Corrigan] new_master {Archie Corrigan}{CNUHuSKySm6osx2heQAIqg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-02-10 12:15:39,173][INFO ][http                     ] [Archie Corrigan] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2016-02-10 12:15:39,173][INFO ][node                     ] [Archie Corrigan] started
[2016-02-10 12:15:39,362][INFO ][gateway                  ] [Archie Corrigan] recovered [0] indices into cluster_state

1 个答案:

答案 0 :(得分：0)

我发现了问题。我必须在输入中提供sincedb_path属性（sincedb_path =＆gt;＆＃34; \ null＆＃34;） - ＆gt;文件属性。因为弹性搜索认为已经创建了此索引，因此无需再次创建它。否则，ES将等待，直到新行添加到日志文件中。

相关问题
最新问题