ActiveMQ:KahaDB和LevelDB减缓磁盘空间扩展

时间:2016-05-16 17:40:36

标签: activemq apache-karaf leveldb apache-servicemix kahadb

是否有人发现他们的ActiveMQ KahaDB或LevelDB商店的磁盘空间正在增长?

我们正在使用以下内容:

  • Windows Server 2008 SP1(用于部署)和Windows 7 SP1(用于开发)
  • Java 1.7
  • ActiveMQ 5.11.1和5.13.2(尝试过KahaDB和LevelDB)
  • ServiceMix 5.4.0,其Karaf服务器托管基于Camel的Java应用程序bean及其未使用的ActiveMQ服务器
  • 消息速率=每秒200条XML消息进入并且大致相同(由Java应用程序bean转换为JSON)
  • 队列数=大约50个队列,其中共享上述消息;特别是2个队列处理高达一半的流量
  • 所有邮件设置为:
    • delivery persistence = false
    • 生存时间(TTL)= 1或2分钟(最长)

KahaDB

当将ActiveMQ与KahaDB一起使用时,我已经看过db * .log文件最初从db-1.log到“db-2.log”再到“db-3.log”之类的“leap-frog”,依此类推正在按预期清除(删除)旧的db * .log文件。然后在某个时刻可能是10个小时之后,旧的db * .log文件停止清理;出现新的db * .log文件,KahaDB开始以每天大约4 GB的速度扩展。当它最终达到我们配置的50 GB上限时,服务器当然会停止工作。

似乎少数消息未能过期(尽管给出了TTL)并因此阻止正常清理。我试图手动删除旧的db * .log文件,但是我被阻止这样做,并警告说这些文件仍在使用中。

KahaDB db * .log文件在文本编辑器中是“半可读”(!);即我可以识别应用程序消息,但这并没有告诉我很多真正发生的事情。

性LevelDB

我最近尝试将LevelDB与ActiveMQ 5.13.2一起使用,并且该商店似乎仍然会扩展得更慢(每天大约200 MB)。

我相信LevelDB数据存储使用压缩;大多数文件看起来都是二进制的,因此没有合适的阅读器/浏览器就会非常不透明。

是否有人发现或实现了读取/浏览KahaDB或LevelDB消息存储库的非持久性消息的工具?

我们的配置

以下是我们的 activemq.xml 通常的样子:

<beans
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.conf}/credentials.properties</value>
        </property>
    </bean>
    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
        lazy-init="false" scope="singleton"
        init-method="start" destroy-method="stop">
    </bean>

    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="XXXX" dataDirectory="${activemq.data}">

        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry topic=">" >
                        <pendingMessageLimitStrategy>
                            <constantPendingMessageLimitStrategy limit="1000"/>
                        </pendingMessageLimitStrategy>
                    </policyEntry>
                    <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
                        <deadLetterStrategy>
                            <discarding /> 
                        </deadLetterStrategy>
                    </policyEntry>
                </policyEntries>
            </policyMap>
        </destinationPolicy>

        <!--
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>
        -->
        <managementContext>
            <managementContext createConnector="true" connectorPort="1099"/>
        </managementContext>

        <!--
        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb"/>
        </persistenceAdapter>
        -->
        <persistenceAdapter>
            <levelDB directory="${activemq.data}/leveldb"/>
        </persistenceAdapter>

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <networkConnectors>
            <networkConnector
                    uri="XXX"
                    networkTTL="XXX"
                    duplex="XXX"
                    prefetchSize="XXX"
                    userName="${networkconnector.username}"
                    password="${networkconnector.password}"
                    name="XXX" >
                <excludedDestinations>
                </excludedDestinations>
                <staticallyIncludedDestinations>
                    <queue physicalName="XXX" /> 
                    <queue physicalName="XXX" />
                </staticallyIncludedDestinations>
            </networkConnector>
        </networkConnectors>

        <transportConnectors>
            <transportConnector name="amqp"              uri="amqp://0.0.0.0:XXXX?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="openwire"          uri="tcp://0.0.0.0:XXXX?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="openwire-outbound" uri="tcp://0.0.0.0:XXXX?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
        </transportConnectors>
        <plugins>
            <simpleAuthenticationPlugin>
                <users>
                    <authenticationUser username="XXX"   password="XXX"   groups="XXX" />
                </users>
            </simpleAuthenticationPlugin>
            <authorizationPlugin>
                <map>
                    <authorizationMap>
                        <authorizationEntries>
                            <authorizationEntry topic=">"                    read="XXX"  write="XXX"  admin="XXX" />
                            <authorizationEntry queue=">"                    read="XXX"  write="XXX"  admin="XXX" />
                            <authorizationEntry queue="XXX"                  read="XXX"  write="XXX"  admin="XXX" />
                            <authorizationEntry queue="ActiveMQ.Advisory.>"  read="XXX"  write="XXX"  admin="XXX" />
                            <authorizationEntry topic="ActiveMQ.Advisory.>"  read="XXX"  write="XXX"  admin="XXX" />
                        </authorizationEntries>
                        <tempDestinationAuthorizationEntry>
                            <tempDestinationAuthorizationEntry read="tempDestinationAdmins" write="tempDestinationAdmins" admin="tempDestinationAdmins" />
                        </tempDestinationAuthorizationEntry>
                    </authorizationMap>
                </map>
            </authorizationPlugin>
            <runtimeConfigurationPlugin checkPeriod="1000" />
        </plugins>
        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>
    </broker>
    <import resource="jetty.xml"/>
</beans>

0 个答案:

没有答案