我检查过这些问题:
Logging from a storm bolt - where is it going?
解决方案不再适用了。
理论上,storm.log.dir
的系统变量在我们启动storm jar
时设置。正如解决方案所示,您可以使用ps aux | grep storm.log.dir
来搜索参数的值。
它显示:
java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/apache-storm-1.1.1 -Dstorm.log.dir=/opt/apache-storm-1.1.1/logs .....
但是,当我去那里时,我有:
[root@xxx ~]# ls -la /opt/apache-storm-1.1.1/logs
total 58272
drwxr-xr-x 3 root root 4096 Jan 15 01:21 .
drwxr-xr-x 14 root root 4096 Oct 31 10:09 ..
-rw-r--r-- 1 root root 0 Nov 3 10:46 access-logviewer.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-nimbus.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-supervisor.log
-rw-r--r-- 1 root root 0 Nov 23 14:36 access-ui.log
-rw-r--r-- 1 root root 8916 Nov 27 17:34 access-web-logviewer.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-web-nimbus.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 access-web-supervisor.log
-rw-r--r-- 1 root root 31661 Nov 27 17:40 access-web-ui.log
-rw-r--r-- 1 root root 2247 Nov 3 11:34 logviewer.log
-rw-r--r-- 1 root root 0 Nov 3 10:46 logviewer.log.metrics
-rw-r--r-- 1 root root 20690 Oct 31 10:09 nimbus.log
-rw-r--r-- 1 root root 0 Oct 31 10:09 nimbus.log.metrics
-rw-r--r-- 1 root root 46713727 Feb 6 17:13 supervisor.log
-rw-r--r-- 1 root root 3143062 Jan 10 08:23 supervisor.log.1.gz
-rw-r--r-- 1 root root 3104009 Jan 11 22:06 supervisor.log.2.gz
-rw-r--r-- 1 root root 3103550 Jan 13 11:43 supervisor.log.3.gz
-rw-r--r-- 1 root root 3103899 Jan 15 01:21 supervisor.log.4.gz
-rw-r--r-- 1 root root 0 Oct 31 10:09 supervisor.log.metrics
-rw-r--r-- 1 root root 401456 Nov 27 17:40 ui.log
-rw-r--r-- 1 root root 0 Nov 23 14:36 ui.log.metrics
drwxr-xr-x 48 root root 4096 Feb 12 18:08 workers-artifacts
输入workers-artifacts
,我有以节点命名的dirs,在里面,我有worker.yaml
。
[root@mq1-acustats-process KafkaStormRadius-1-1518455061]# cd 1027
[root@mq1-acustats-process 1027]# ls -la
total 4
drwxr-xr-x 2 root root 24 Feb 12 18:04 .
drwxr-xr-x 8 root root 72 Feb 12 18:04 ..
-rw-r--r-- 1 root root 109 Feb 12 18:04 worker.yaml
[root@mq1-acustats-process 1027]# cat worker.yaml
worker-id: 33e6e77d-8a7b-48b2-b2d4-1be0af726c52
logs.users: []
logs.groups: []
topology.submitter.user: root
所以,这里没有日志。他们在哪儿?
我在worker.xml
下面有{STORM_DIR}/log4j2
:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration monitorInterval="60" shutdownHook="disable">
<properties>
<property name="pattern">%d{yyyy-MM-dd HH:mm:ss.SSS} %c{1.} %t [%p] %msg%n</property>
<property name="patternNoTime">%msg%n</property>
<property name="patternMetrics">%d %-8r %m%n</property>
</properties>
<appenders>
<RollingFile name="A1"
fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
<PatternLayout>
<pattern>${pattern}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
<RollingFile name="radius"
fileName="${sys:storm.log.dir}/radius.log"
filePattern="${sys:storm.log.dir}/radius.log.%i.gz">
<PatternLayout>
<pattern>${pattern}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 20 MB -->
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
<RollingFile name="STDOUT"
fileName="${sys:storm.log.dir}/radius.out"
filePattern="${sys:storm.log.dir}/radius.out.%i.gz">
<PatternLayout>
<pattern>${patternNoTime}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="4"/>
</RollingFile>
<RollingFile name="STDERR"
fileName="${sys:storm.log.dir}/radius.err"
filePattern="${sys:storm.log.dir}/radius.err.%i.gz">
<PatternLayout>
<pattern>${patternNoTime}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="4"/>
</RollingFile>
<RollingFile name="METRICS"
fileName="${sys:storm.log.dir}/radius.metrics"
filePattern="${sys:storm.log.dir}/radius.metrics.%i.gz">
<PatternLayout>
<pattern>${patternMetrics}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="2 MB"/>
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
<Syslog name="syslog" format="RFC5424" charset="UTF-8" host="localhost" port="514"
protocol="UDP" appName="[${sys:storm.id}:${sys:worker.port}]" mdcId="mdc" includeMDC="true"
facility="LOCAL5" enterpriseNumber="18060" newLine="true" exceptionPattern="%rEx{full}"
messageId="[${sys:user.name}:${sys:logging.sensitivity}]" id="storm" immediateFail="true" immediateFlush="true"/>
</appenders>
<loggers>
<root level="info"> <!-- We log everything -->
<appender-ref ref="A1"/>
<appender-ref ref="syslog"/>
</root>
<Logger name="org.apache.storm.metric.LoggingMetricsConsumer" level="info" additivity="false">
<appender-ref ref="METRICS"/>
</Logger>
<Logger name="com.joestelmach.natty" level="error" additivity="false">
<appender-ref ref="radius"/>
</Logger>
<Logger name="STDERR" level="INFO">
<appender-ref ref="STDERR"/>
<appender-ref ref="syslog"/>
</Logger>
<Logger name="STDOUT" level="INFO">
<appender-ref ref="STDOUT"/>
<appender-ref ref="syslog"/>
</Logger>
</loggers>
</configuration>
但是,我不会在radius.log
下看到${sys:storm.log.dir}
。
为什么呢?
顺便说一下:
cluster.xml
和worker.xml
之间的区别?答案 0 :(得分:1)
我的猜测是,没有为worker JVM正确设置storm.log.dir变量。请记住,在运行storm jar
时启动的JVM与运行拓扑的JVM不同。
为了为您的worker设置VM选项,您可以使用storm.yaml中的worker.childopts变量来全局设置选项(如果您这样做,您可能希望确保复制https://github.com/apache/storm/blob/v1.1.1/conf/defaults.yaml#L171的默认值这个)或拓扑配置中的topology.worker.childopts为每个拓扑设置它们。
例如,我使用以下配置将日志打印到E:\ testLogs:
storm.yaml
worker.childopts: "-Dstorm.log.dir=E:\\testLogs"
worker.xml
...
<RollingFile name="A1"
fileName="${sys:storm.log.dir}/worker.log"
filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
<PatternLayout>
<pattern>${pattern}</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
</Policies>
<DefaultRolloverStrategy max="9"/>
</RollingFile>
...
其中,worker.xml的其余部分是与Storm 1.1.1一起发布的默认值。
关于cluster.xml和worker.xml,cluster.xml是Storm守护程序(Nimbus,Supervisor,UI等)的日志配置,而worker.xml是工作进程(运行您的进程)的日志配置拓扑组件)
答案 1 :(得分:0)
我已经阅读了另一个团队留下的文档并找到了这些行....我有一个风暴服务没有启动。
systemctl start storm-supervisor
检查/etc/systemd/system/
下的服务定义(我使用的是CentOS,因此我使用的是systemd
),它显示为:
[Unit]
Description=Nimbus Service
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=0s
ExecStart=/opt/apache-storm/bin/storm supervisor
[Install]
WantedBy=multi-user.target
启动后,我现在看到在正确目录下创建的日志文件。
值得一提的是,我还改变了另外两件事:
slf4j
代替log4j
。log4j2.properties
以使用绝对路径(在/tmp
下),而不是像{sys:storm.log.dir}
那样读取系统变量,以便丢弃可变因子。