Question

我们已经设置了一个Spring Batch批量电子邮件发件人作业，该作业执行以下操作：

从JMS队列中读取传入的电子邮件详细信息。
不要做任何处理，只需通过它们......
将详细信息转换为电子邮件并将其“写入”SMTP
无限重复广告

这通常可以正常工作，但我们发现了大量的内存泄漏。大约9个小时后，可以看到一个FlowJob，但是748 JobExecution（所有'STARTED'），每个持有778（！）StepExecution个实例。总而言之，900 MB的东西。

这是Spring配置（Spring 3.1，Spring Batch 2.1.9）：

<bean id="jobRepository"
    class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />

<bean id="jobLauncher"
    class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
    <property name="taskExecutor">
        <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor"/>
    </property>
</bean>

<bean id="jmsEmailFetcher" class="org.springframework.batch.item.jms.JmsItemReader">
    <property name="jmsTemplate" ref="batchEmailJmsTemplate" />
</bean>

<bean id="passthroughProcessor" class="org.springframework.batch.item.support.PassThroughItemProcessor" />

<bean id="transactionManager"
    class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

<!-- The Spring Batch *Limiter/Decider* -->

<bean id="ourLimitDecider" class="our.special.InfiniteThrottledExecutionDecider" />

<!-- The Spring Batch *Job* -->

<batch:job id="fetchEmailsJobBatch" restartable="true">
    <batch:step id="fetchEmailsStep" next="limitDecision">
        <batch:tasklet throttle-limit="10">
            <batch:chunk reader="jmsEmailFetcher" processor="passthroughProcessor"
                         writer="batchEmailService.javaMailItemWriter" commit-interval="100" skip-limit="999999999">  <!-- Might gets *lots* of MQ-not-up-yet fails -->
                <batch:skippable-exception-classes>
                    <batch:include class="org.springframework.jms.JmsException" />       <!-- Generally MQ-not-up-yet ConnectionException, or Session-closed (in tests) -->
                    <batch:include class="java.lang.IllegalStateException" />            <!-- Yuk, usually/presumably a test SMTP server isn't yet connected -->
                    <batch:include class="org.springframework.mail.MailSendException" /> <!-- SMTP down... -->
                </batch:skippable-exception-classes>
            </batch:chunk>
        </batch:tasklet>
    </batch:step>
    <batch:decision id="limitDecision" decider="ourLimitDecider">
        <batch:next on="CONTINUE" to="fetchEmailsStep" />
        <batch:end on="COMPLETED" />
    </batch:decision>
</batch:job>

我们的InfiniteThrottledExecutionDecider每次基本上都返回new FlowExecutionStatus("CONTINUE")，以确保fetchEmailsStep在流程结束时执行，并且作业永远不会完成 - 至少在我们之前准备好自己阻止它。

我们不使用数据库，因此我们希望在内存中保留一些内容，但不要使用已经运行的所有内容的完整记录 ...

我们的配置有问题吗？还是我们的做法？

我们的日志文件中有一点是我们被告知的步骤＃778以及应该是唯一的Job实例。

23:58:18,782 - INFO  (org.springframework.batch.core.job.SimpleStepHandler) - Duplicate step [fetchEmailsStep] detected in execution of job=[fetchEmailsJobBatch]. If either step fails, both will be executed again on restart.
23:59:52,257 - INFO  (org.springframework.batch.core.job.SimpleStepHandler) - Executing step: [fetchEmailsStep]
23:59:52,257 - DEBUG (org.springframework.batch.core.step.AbstractStep) - Executing: id=778
23:59:52,259 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Starting repeat context.
23:59:52,259 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat operation about to start at count=1
23:59:52,259 - DEBUG (org.springframework.batch.core.scope.context.StepContextRepeatCallback) - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@1be1ee
23:59:52,259 - DEBUG (org.springframework.batch.core.scope.context.StepContextRepeatCallback) - Chunk execution starting: queue size=0
23:59:52,259 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Starting repeat context.
23:59:52,259 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat operation about to start at count=1
... 5 second JMS timeout ...
23:59:57,716 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat is complete according to policy and result value.
23:59:57,716 - DEBUG (org.springframework.batch.core.step.item.ChunkOrientedTasklet) - Inputs not busy, ended: true
23:59:57,716 - DEBUG (org.springframework.batch.core.step.tasklet.TaskletStep) - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
23:59:57,719 - DEBUG (org.springframework.batch.core.step.tasklet.TaskletStep) - Saving step execution before commit: StepExecution: id=778, version=1, name=fetchEmailsStep, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
23:59:57,721 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat is complete according to policy and result value.
23:59:57,721 - DEBUG (org.springframework.batch.core.step.AbstractStep) - Step execution success: id=778
23:59:57,722 - DEBUG (org.springframework.batch.core.step.AbstractStep) - Step execution complete: StepExecution: id=778, version=3, name=fetchEmailsStep, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0
23:59:57,723 - DEBUG (org.springframework.batch.core.job.flow.support.SimpleFlow) - Completed state=fetchEmailsJobBatch.fetchEmailsStep with status=COMPLETED
23:59:57,723 - DEBUG (org.springframework.batch.core.job.flow.support.SimpleFlow) - Handling state=fetchEmailsJobBatch.limitDecision100
23:59:57,723 - DEBUG (org.springframework.batch.core.job.flow.support.SimpleFlow) - Completed state=fetchEmailsJobBatch.limitDecision100 with status=CONTINUE
23:59:57,723 - DEBUG (org.springframework.batch.core.job.flow.support.SimpleFlow) - Handling state=fetchEmailsJobBatch.fetchEmailsStep

问题是，堆转储显示748 JobExecution s和581,911 StepExecution s。它们都来自哪里？

Answer 1

好的，在回答我的问题时，解决办法就是放弃Step Flow并让“fetchEmailsStep”永远运行。

我们通过删除JmsTemplate（“batchEmailJmsTemplate”）的“readTimeout”并继承org.springframework.batch.item.jms.JmsItemReader以删除检查来阻止您使用无限JmsTemplate来实现这一点 - 并添加额外的日志记录如果您不想覆盖，可以将“readTimeout”设置为一个巨大的数字。

此处有更多讨论：http://forum.springsource.org/showthread.php?132468-Massive-memory-use-with-FlowJob-repeating-Steps&p=431279

我仍然不知道为什么内存使用量和它一样大......

[编辑] 我还使用JobExecutionDecider（“our.special.InfiniteThrottledExecutionDecider”）来实现限制（基于<StepExecution>.writeCount），但这必须走。我们现在使用扩展StepExecutionListenerSupport的新类（或者您可以实现StepExecutionListener）来实现相同的结果。此bean的一个实例作为<batch:listener>添加到我们的单个<batch:step>

使用FlowJob进行Spring Batch内存泄漏，重复步骤

1 个答案: