主配置

Question

以下是我们正在努力实现的目标。

我们希望在不同的vms中并行地在数据库中暂存一个大的xml文件。为实现这一目标，我们正在使用可扩展的Spring批量远程分区方法，我们遇到了一些问题。以下是高级设置

master - 将xml文件拆分为多个分区（我们当前的网格大小为3）。
slave 1 - 处理分区（读取基于索引的分区并写入DB）
slave 2 - 处理分区

我们在Linux中使用活动MQ 5.15.3运行它。

使用上述设置

slave 1同时处理2个分区
slave 2正在处理1个分区。

主人没有等待所有奴隶完成并且将进入未知状态。

org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)

如果我们的网格大小为2，那么从属1正在挑选它们而奴隶2没有得到它们中的任何一个（因此贪婪的奴隶）。从属1正在并行处理它们，但作业将进入未知状态。

以下是我们的问题

我们如何防止slave 1并行处理两个分区 - 我们尝试将prefetch设置为0和1 [链接] http://activemq.apache.org/what-is-the-prefetch-limit-for.html [链接]，但这不起作用。我们怎样才能让奴隶一次处理一个分区？
为什么主人不等待所有的奴隶？

以下是我们的配置

主配置

 <?xml version="1.0" encoding="UTF-8"?>
    <step id="remotePartitionStagingStep">
     <partition partitioner="xmlPartitioner" handler="partitionStagingHandler"/>
       <listeners>
         <listener ref="jobListener" />
       </listeners>
     </step>
     <!-- XML Partitioner starts here -->
     <beans:bean id="xmlPartitioner" class="XMLPartitioner" scope="step">           <beans:property name="partitionThreadName" value="ImportXMLPartition-"/>
      <beans:property name="resource" value="file:///#{jobParameters[ImportFileProcessed]}"/>
            <beans:property name="rootElementNode" value="#{jobParameters[ImportFragmentRootNameWithPrefix]}"/>
        </beans:bean>
       <beans:bean id="partitionStagingHandler" class="org.springframework.batch.integration.partition.MessageChannelPartitionHandler">
            <beans:property name="stepName" value="slave.ImportStagingStep"/>
            <beans:property name="gridSize" value="3"/>
            <beans:property name="replyChannel" ref="aggregatedReplyChannel"/>
            <beans:property name="jobExplorer" ref="jobExplorer"/>
            <beans:property name="messagingOperations">
                <beans:bean class="org.springframework.integration.core.MessagingTemplate">
                    <beans:property name="defaultChannel" ref="requestsChannel"/>
                    <beans:property name="receiveTimeout" value="${batch.gateway.receiveTimeout}" />  //360000 is the current value
                </beans:bean>
            </beans:property>
        </beans:bean>
         <int:aggregator ref="partitionStagingHandler"
                        input-channel="replyChannel"
                        output-channel="aggregatedReplyChannel"
                        send-timeout="${batch.gateway.receiveTimeout}"
                        expire-groups-upon-timeout="true"/>
         <int:channel id="requestsChannel">
            <int:interceptors>
                <int:wire-tap channel="logChannel"/>
            </int:interceptors>
        </int:channel>
         <int-jms:outbound-channel-adapter connection-factory="connectionFactory"
                                          channel="requestsChannel"
                                          destination-name="requestsQueue"/>
        <int:channel id="aggregatedReplyChannel">
            <int:queue/>
        </int:channel>
         <int:channel id="replyChannel">
            <int:interceptors>
                <int:wire-tap channel="logChannel"/>
            </int:interceptors>
        </int:channel>
         <int-jms:message-driven-channel-adapter connection-factory="connectionFactory"
                                                channel="replyChannel"
                                                error-channel="errorChannel"
                                                destination-name="replyQueue"/>

从属配置

    <step id="slave.ImportStagingStep">
    <tasklet transaction-manager="transactionManager">
    <chunk reader="StagingSpecificItemReader" processor="chainedStagingProcessor" writer="StagingItemWriter"
             commit-interval="${import.CommitInterval}" skip-limit="${import.skipLimit}" retry-policy="neverRetryPolicy">
     <streams>
      <stream ref="errorFlatFileRecordWriter"/>
     </streams>
     <skippable-exception-classes>
       <include class="java.lang.Exception"/>
       <exclude class="org.springframework.oxm.UnmarshallingFailureException"/>
       <exclude class="java.lang.Error"/>
     </skippable-exception-classes>
     </chunk>
     <listeners>
        <listener ref="stepExceptionListener"/>
        <listener ref="stagingListener"/>
     </listeners>
       </tasklet>
       <listeners>
          <listener ref="slaveStepExecutionListener"/>
       </listeners>
      </step>
        <beans:bean id="slaveStepExecutionListener" class="StepExecutionListener"></beans:bean>
        <!-- JMS config for staging step starts here -->
        <int:channel id="replyChannel">
            <int:interceptors>
                <int:wire-tap channel="logChannel"/>
            </int:interceptors>
        </int:channel>
            <int:channel id="requestChannel">
            <int:interceptors>
                <int:wire-tap channel="logChannel"/>
            </int:interceptors>
        </int:channel>
        <int-jms:message-driven-channel-adapter connection-factory="connectionFactory"
                                                destination-name="requestsQueue"
                                                error-channel="errorChannel"
                                                channel="requestsChannel"/>

        <int-jms:outbound-channel-adapter connection-factory="connectionFactory"
                                          destination-name="replyQueue"
                                          channel="replyChannel"/>

        <int:service-activator input-channel="requestsChannel"
                               output-channel="replyChannel"
                               ref="stepExecutionRequestHandler"/>

        <!-- JMS config for staging step ends here -->

        <!-- The logChannel is configured as an interceptor to channels so that messages are logged. -->
        <int:logging-channel-adapter auto-startup="true" log-full-message="true" id="logChannel" level="INFO"/>

        <int:channel id="errorChannel" />

        <int:service-activator input-channel="errorChannel" method="handleException">
          <beans:bean class="ErrorHandler" />
        </int:service-activator>

Answer 1

找到了我们遇到的问题。

奴隶包括弹簧批次作业定义两次，导致每个奴隶的消费者数量被登记两次。我认为当我们在从属设备上同时运行两个分区时，我们缺少一些配置来将消息从从设备路由回主设备，所以只要我们有一个分区转到一个从设备（我们是多线程的奴隶）我们没事。

远程分区 - 奴隶变得贪婪

主配置

从属配置

1 个答案: