Apache Storm 1.1.1 - 在哪里可以找到我的日志文件?

时间:2018-02-12 17:29:56

标签: log4j2 apache-storm

我检查过这些问题:

Logging from a storm bolt - where is it going?

解决方案不再适用了。

理论上,storm.log.dir的系统变量在我们启动storm jar时设置。正如解决方案所示,您可以使用ps aux | grep storm.log.dir来搜索参数的值。

它显示:

java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/apache-storm-1.1.1 -Dstorm.log.dir=/opt/apache-storm-1.1.1/logs .....

但是,当我去那里时,我有:

[root@xxx ~]# ls -la /opt/apache-storm-1.1.1/logs
total 58272
drwxr-xr-x  3 root root     4096 Jan 15 01:21 .
drwxr-xr-x 14 root root     4096 Oct 31 10:09 ..
-rw-r--r--  1 root root        0 Nov  3 10:46 access-logviewer.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-nimbus.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-supervisor.log
-rw-r--r--  1 root root        0 Nov 23 14:36 access-ui.log
-rw-r--r--  1 root root     8916 Nov 27 17:34 access-web-logviewer.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-web-nimbus.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-web-supervisor.log
-rw-r--r--  1 root root    31661 Nov 27 17:40 access-web-ui.log
-rw-r--r--  1 root root     2247 Nov  3 11:34 logviewer.log
-rw-r--r--  1 root root        0 Nov  3 10:46 logviewer.log.metrics
-rw-r--r--  1 root root    20690 Oct 31 10:09 nimbus.log
-rw-r--r--  1 root root        0 Oct 31 10:09 nimbus.log.metrics
-rw-r--r--  1 root root 46713727 Feb  6 17:13 supervisor.log
-rw-r--r--  1 root root  3143062 Jan 10 08:23 supervisor.log.1.gz
-rw-r--r--  1 root root  3104009 Jan 11 22:06 supervisor.log.2.gz
-rw-r--r--  1 root root  3103550 Jan 13 11:43 supervisor.log.3.gz
-rw-r--r--  1 root root  3103899 Jan 15 01:21 supervisor.log.4.gz
-rw-r--r--  1 root root        0 Oct 31 10:09 supervisor.log.metrics
-rw-r--r--  1 root root   401456 Nov 27 17:40 ui.log
-rw-r--r--  1 root root        0 Nov 23 14:36 ui.log.metrics
drwxr-xr-x 48 root root     4096 Feb 12 18:08 workers-artifacts

输入workers-artifacts,我有以节点命名的dirs,在里面,我有worker.yaml

[root@mq1-acustats-process KafkaStormRadius-1-1518455061]# cd 1027

[root@mq1-acustats-process 1027]# ls -la
total 4
drwxr-xr-x 2 root root  24 Feb 12 18:04 .
drwxr-xr-x 8 root root  72 Feb 12 18:04 ..
-rw-r--r-- 1 root root 109 Feb 12 18:04 worker.yaml

[root@mq1-acustats-process 1027]# cat worker.yaml
worker-id: 33e6e77d-8a7b-48b2-b2d4-1be0af726c52
logs.users: []
logs.groups: []
topology.submitter.user: root

所以,这里没有日志。他们在哪儿?

我在worker.xml下面有{STORM_DIR}/log4j2

<?xml version="1.0" encoding="UTF-8"?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->

<configuration monitorInterval="60" shutdownHook="disable">
<properties>
    <property name="pattern">%d{yyyy-MM-dd HH:mm:ss.SSS} %c{1.} %t [%p] %msg%n</property>
    <property name="patternNoTime">%msg%n</property>
    <property name="patternMetrics">%d %-8r %m%n</property>
</properties>
<appenders>
    <RollingFile name="A1"
        fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
        filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
        </Policies>
        <DefaultRolloverStrategy max="9"/>
    </RollingFile>
    <RollingFile name="radius"
        fileName="${sys:storm.log.dir}/radius.log"
        filePattern="${sys:storm.log.dir}/radius.log.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 20 MB -->
        </Policies>
        <DefaultRolloverStrategy max="9"/>
    </RollingFile>
    <RollingFile name="STDOUT"
        fileName="${sys:storm.log.dir}/radius.out"
        filePattern="${sys:storm.log.dir}/radius.out.%i.gz">
        <PatternLayout>
            <pattern>${patternNoTime}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
        </Policies>
        <DefaultRolloverStrategy max="4"/>
    </RollingFile>
    <RollingFile name="STDERR"
        fileName="${sys:storm.log.dir}/radius.err"
        filePattern="${sys:storm.log.dir}/radius.err.%i.gz">
        <PatternLayout>
            <pattern>${patternNoTime}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
        </Policies>
        <DefaultRolloverStrategy max="4"/>
    </RollingFile>
    <RollingFile name="METRICS"
        fileName="${sys:storm.log.dir}/radius.metrics"
        filePattern="${sys:storm.log.dir}/radius.metrics.%i.gz">
        <PatternLayout>
            <pattern>${patternMetrics}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="2 MB"/>
        </Policies>
        <DefaultRolloverStrategy max="9"/>
    </RollingFile>
    <Syslog name="syslog" format="RFC5424" charset="UTF-8" host="localhost" port="514"
        protocol="UDP" appName="[${sys:storm.id}:${sys:worker.port}]" mdcId="mdc" includeMDC="true"
        facility="LOCAL5" enterpriseNumber="18060" newLine="true" exceptionPattern="%rEx{full}"
        messageId="[${sys:user.name}:${sys:logging.sensitivity}]" id="storm" immediateFail="true" immediateFlush="true"/>
</appenders>
<loggers>
    <root level="info"> <!-- We log everything -->
        <appender-ref ref="A1"/>
        <appender-ref ref="syslog"/>
    </root>
    <Logger name="org.apache.storm.metric.LoggingMetricsConsumer" level="info" additivity="false">
        <appender-ref ref="METRICS"/>
    </Logger>
    <Logger name="com.joestelmach.natty" level="error" additivity="false">
        <appender-ref ref="radius"/>
    </Logger>
    <Logger name="STDERR" level="INFO">
        <appender-ref ref="STDERR"/>
        <appender-ref ref="syslog"/>
    </Logger>
    <Logger name="STDOUT" level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="syslog"/>
    </Logger>
</loggers>
</configuration>

但是,我不会在radius.log下看到${sys:storm.log.dir}

为什么呢?

顺便说一下:

  • cluster.xmlworker.xml之间的区别?

2 个答案:

答案 0 :(得分:1)

我的猜测是,没有为worker JVM正确设置storm.log.dir变量。请记住,在运行storm jar时启动的JVM与运行拓扑的JVM不同。

为了为您的worker设置VM选项,您可以使用storm.yaml中的worker.childopts变量来全局设置选项(如果您这样做,您可能希望确保复制https://github.com/apache/storm/blob/v1.1.1/conf/defaults.yaml#L171的默认值这个)或拓扑配置中的topology.worker.childopts为每个拓扑设置它们。

例如,我使用以下配置将日志打印到E:\ testLogs:

storm.yaml

worker.childopts: "-Dstorm.log.dir=E:\\testLogs"

worker.xml

...
<RollingFile name="A1"
    fileName="${sys:storm.log.dir}/worker.log"
    filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
    <PatternLayout>
        <pattern>${pattern}</pattern>
    </PatternLayout>
    <Policies>
        <SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
    </Policies>
    <DefaultRolloverStrategy max="9"/>
</RollingFile>
...

其中,worker.xml的其余部分是与Storm 1.1.1一起发布的默认值。

关于cluster.xml和worker.xml,cluster.xml是Storm守护程序(Nimbus,Supervisor,UI等)的日志配置,而worker.xml是工作进程(运行您的进程)的日志配置拓扑组件)

答案 1 :(得分:0)

我已经阅读了另一个团队留下的文档并找到了这些行....我有一个风暴服务没有启动。

systemctl start storm-supervisor

检查/etc/systemd/system/下的服务定义(我使用的是CentOS,因此我使用的是systemd),它显示为:

[Unit]
Description=Nimbus Service
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=0s
ExecStart=/opt/apache-storm/bin/storm supervisor

[Install]
WantedBy=multi-user.target

启动后,我现在看到在正确目录下创建的日志文件。

值得一提的是,我还改变了另外两件事:

  • 使用slf4j代替log4j
  • 更改log4j2.properties以使用绝对路径(在/tmp下),而不是像{sys:storm.log.dir}那样读取系统变量,以便丢弃可变因子。