SpringBatch重复调用项目上的处理器

时间:2016-06-23 15:42:33

标签: java spring spring-batch batch-processing bean-io

我们正在使用Spring Batch将管道分隔文件ETL到DB中。文件中的每条记录都有许多字段,并由ClaimNumber标识:

ClaimNumber|AdjustmentVersion|.....
0038017282|3|....
0071517729|3|....
0081517745|3|....

在批处理步骤中,使用常规的读取 - 处理 - 写入流程:

<step id="stagingDataDump" next="gatherStats">
            <tasklet>
                <!-- <chunk reader="genericBatchItemReader" writer="genericBatchItemWriter" -->
                <chunk reader="genericBatchItemReader" writer="compositeWriter" processor="validationProcessor"
                    commit-interval="1000" skip-limit="100000" > 
                    <skippable-exception-classes>
                        <batch:include  class="org.springframework.batch.item.file.FlatFileParseException" />
                        <batch:include  class="org.beanio.BeanIOException" />
                    </skippable-exception-classes>
                </chunk>
                <listeners>
                    <listener ref="genericItemSkipListener"/>
                </listeners>
            </tasklet>
        </step>

Reader使用BeanIOFlatFileItemReader

<bean id="genericBatchItemReader"   class="org.beanio.spring.BeanIOFlatFileItemReader"  scope="step"
        p:streamMapping="classpath:beanio-mapping.xml" 
        p:streamName="#{jobParameters[feedProcessorLauncherImpl.BEANIO_STREAM_MAPPING]}" 
        p:resource="file://#{jobParameters[feedProcessorLauncherImpl.RESOURCE_FILE_NAME_UNENCRYPTED]}" 
        p:errorHandler-ref="beanIoRecordErrorHandler"/>

处理器阶段封装了项目验证:

<util:map id="handlerRegistryContents">
        <entry key="#{T(org.fuwt.iws.claims.service.filemanagement.ClaimsEnums$ContentSubType).MEDI}" value-ref="medicalClaimsValidator"/>
        <entry key="#{T(org.fuwt.iws.claims.service.filemanagement.ClaimsEnums$ContentSubType).LAB}" value-ref="labClaimsValidator"/>
        <entry key="#{T(org.fuwt.iws.claims.service.filemanagement.ClaimsEnums$ContentSubType).RXPD}" value-ref="pharmaClaimsValidator"/>
</util:map>

<bean id="validationProcessor"  class="org.fuwt.iws.claims.validation.springbatch.ValidationProcessor"  scope="step">

    <property name="handlerRegistry" ref="handlerRegistryContents"/>

</bean> 

写是复合的:

    <bean id="genericBatchItemWriter"   class="org.fuwt.iws.claims.springbatch.GenericBatchItemWriter"  scope="step"
        p:metadataId="#{jobParameters[feedProcessorLauncherImpl.METADATA_ID]}"/>

    <bean id="softValidationsItemWriter"    class="org.fuwt.iws.claims.springbatch.SoftValidationsItemWriter"   scope="step"
        p:metadataId="#{jobParameters[feedProcessorLauncherImpl.METADATA_ID]}"/>

<bean id="compositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter" scope="step">
        <property name="delegates">
            <list>
                <!-- Order here is significant as ID's, which are generated by the first writer - genericBatchItemWriter - need to be passed around -->
                <ref bean="genericBatchItemWriter"/>
                <ref bean="softValidationsItemWriter"/>
            </list>
        </property>
    </bean> 

在上面的处理/验证步骤中,ValidationProcessor确定记录的类型,根据该类型实例化适当的复合验证器(在这种情况下为MedicalClaimsValidator),其中所有个体在下面的日志中配置了此类型的验证(复合模式),HCPCSCodeLength

虽然通过此基础架构验证了每个项目(声明),但发现的错误会累积到项目errors字段中 - Map<String, Collection<String>>,其中特定验证的失败由此键入验证名称并在地图的值中进行了描述。

这让我想到了我们所看到的错误行为:

在仔细检查上述测试CSV文件的负载后,对3个索赔记录进行了日志文件检查,我们发现了以下(奇怪的!?)行为,即每条记录的验证次数与文件中的序号一样多。如下所示:第一个记录(声明)被验证一次,导致错误映射中的单个记录;第二个记录(声明)被验证两次,它的错误映射现在包含重复两次的消息;第3条记录正在被验证3次,导致进入错误地图,同样的信息重复3次。

文件中的每条记录在无效性方面几乎完全相同,因此预期的结果是每条记录应该具有相同的错误集合。

实际结果是错误的值随着每个后续记录而不断增加:

第一记录:

INFO  2016-06-23 10:16:24,214 [main] org.fuwt.iws.claims.validation.springbatch.medical.MedicalClaimsValidator: Service date to: Thu Dec 10 00:00:00 EST 2015
INFO  2016-06-23 10:16:24,216 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Started HCPCSCodeLength validation on org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength@1a9ddcb7 for claimNumber = 0038017282
INFO  2016-06-23 10:16:24,223 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Errors: {HCPCSCode=[supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value]} 

第二记录

INFO  2016-06-23 10:16:24,227 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Started HCPCSCodeLength validation on org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength@1a9ddcb7 for claimNumber = 0071517729
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Errors: {HCPCSCode=[supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value]}
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Started HCPCSCodeLength validation on org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength@1a9ddcb7 for claimNumber = 0071517729
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Errors: {HCPCSCode=[supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value, supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value]}

第3记录

INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Started HCPCSCodeLength validation on org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength@1a9ddcb7 for claimNumber = 0081517745
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Errors: {HCPCSCode=[supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value]}
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Started HCPCSCodeLength validation on org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength@1a9ddcb7 for claimNumber = 0081517745
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Errors: {HCPCSCode=[supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value, supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value]}
INFO  2016-06-23 10:16:24,228 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Started HCPCSCodeLength validation on org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength@1a9ddcb7 for claimNumber = 0081517745
INFO  2016-06-23 10:16:24,229 [main] org.fuwt.iws.claims.validation.springbatch.medical.HCPCSCodeLength: Errors: {HCPCSCode=[supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value, supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value, supplied HCPCS code is blank or null, but should be 3-5 characters in length, anything greater than or less than is invalid value]}

使用的版本:

spring-batch-core: 2.2.0.RELEASE
beanio: 2.1.0

问题:

如果有什么东西让Spring Batch在处理器上发出那些重复的调用,那么Spring Batch的正常行为是什么,以及如何阻止它并实现我上面描述的所需行为呢?

更新

此验证组件表现出错误的行为:

@Component("medicalClaimsValidator")
public class MedicalClaimsValidator implements ClaimValidation {

    private final static Logger logger = LoggerFactory.getLogger(MedicalClaimsValidator.class);

    @Autowired private AbstractMedicalClaimValidation  HCPCSCodeLength;


    List<ClaimValidation> medicalClaimValidations = new ArrayList<>();

    @Override
    public boolean supports(Class<?> clazz) {
        return QualcareMedicalClaimWeeklyNDT.class.equals(clazz);
    }

    @Override

    public Map<String, Collection<String>> validate(Object item,  MessageSource messageSource) {

        logger.info("\nSoft-validating the bean...");
        QualcareMedicalClaimWeeklyNDT medicalClaim = (QualcareMedicalClaimWeeklyNDT)item;

        logger.info("Claim #: {}", medicalClaim.getClaimNumber());
        logger.info("Service date from: {}", medicalClaim.getServiceDateFrom());
        logger.info("Service date to: {}", medicalClaim.getServiceDateTo());

        //TODO: A candidate for externalization into a config file once we have all the known rules
        //medicalClaimValidations.add(new ServiceDateFromGreaterThanTo());
        //medicalClaimValidations.add(new ProcedureCodeLength());
        medicalClaimValidations.add(HCPCSCodeLength/*new HCPCSCodeLength()*/);
        //medicalClaimValidations.add(new TypeOfBillPresenseAndLengthForInstitutionalClaims());
        //medicalClaimValidations.add(new DischargeStatusPresenseAndLengthForInpatientClaims());
        //medicalClaimValidations.add(new DiagnosisCodeFormat());


        for(ClaimValidation validation:medicalClaimValidations) {
            logger.info("validation type: {}",validation.getClass());
             validation.validate(medicalClaim, messageSource);
        }

        return medicalClaim.getErrors();
    }

}

以下解决方法隐藏了错误行为:

@Component("medicalClaimsValidator")
public class MedicalClaimsValidator implements ClaimValidation {

    private final static Logger logger = LoggerFactory.getLogger(MedicalClaimsValidator.class);

    @Autowired @Qualifier("HCPCSCodeLength")private AbstractMedicalClaimValidation  HCPCSCodeLength;
    @Autowired @Qualifier("serviceDateFromGreaterThanTo")private AbstractMedicalClaimValidation  serviceDateFromGreaterThanTo;
    @Autowired @Qualifier("procedureCodeLength")private AbstractMedicalClaimValidation  procedureCodeLength;
    @Autowired @Qualifier("typeOfBillPresenseAndLengthForInstitutionalClaims")private AbstractMedicalClaimValidation  typeOfBillPresenseAndLengthForInstitutionalClaims;
    @Autowired @Qualifier("dischargeStatusPresenseAndLengthForInpatientClaims")private AbstractMedicalClaimValidation  dischargeStatusPresenseAndLengthForInpatientClaims;
    @Autowired @Qualifier("diagnosisCodeFormat")private AbstractMedicalClaimValidation  diagnosisCodeFormat;


    List<ValidationProcessTuple> medicalClaimValidations = new ArrayList<>();

    @Override
    public boolean supports(Class<?> clazz) {
        return QualcareMedicalClaimWeeklyNDT.class.equals(clazz);
    }

    @Override

    public Map<String, Collection<String>> validate(Object item,  MessageSource messageSource) {

        logger.info("\nSoft-validating the bean...");
        QualcareMedicalClaimWeeklyNDT medicalClaim = (QualcareMedicalClaimWeeklyNDT)item;

        logger.info("Claim #: {}", medicalClaim.getClaimNumber());
        logger.info("Service date from: {}", medicalClaim.getServiceDateFrom());
        logger.info("Service date to: {}", medicalClaim.getServiceDateTo());

        //TODO: A candidate for externalization into a config file once we have all the known rules
        medicalClaimValidations.add(new ValidationProcessTuple(serviceDateFromGreaterThanTo, false));
        medicalClaimValidations.add(new ValidationProcessTuple(procedureCodeLength, false));
        medicalClaimValidations.add(new ValidationProcessTuple(HCPCSCodeLength, false));
        medicalClaimValidations.add(new ValidationProcessTuple(typeOfBillPresenseAndLengthForInstitutionalClaims, false));
        medicalClaimValidations.add(new ValidationProcessTuple(dischargeStatusPresenseAndLengthForInpatientClaims, false));
        medicalClaimValidations.add(new ValidationProcessTuple(diagnosisCodeFormat, false));

        for (ValidationProcessTuple tuple : medicalClaimValidations) {
            if (!tuple.processed) {//to counteract the erroneous behavior whereby validation calls get repeated as many times as there are records
                tuple.validation.validate(item, messageSource);
                tuple.processed = true;
            }
        }

        return medicalClaim.getErrors();
    }

}

我仍然处于黑暗中,为什么这种行为首先发生 - 对此的任何解释当然都是受欢迎的。

1 个答案:

答案 0 :(得分:2)

在MedicalClaimsValidator类中,为什么要在validate()方法中累积此List中的读取项?我没有看到原因。每次处理新行时,它都会在此列表中不断添加已处理的元组。您可以在方法,init方法或构造函数中定义处理规则。

   //TODO: A candidate for externalization into a config file once we have all the known rules
    medicalClaimValidations.add(new ValidationProcessTuple(serviceDateFromGreaterThanTo, false));
    medicalClaimValidations.add(new ValidationProcessTuple(procedureCodeLength, false));
    medicalClaimValidations.add(new ValidationProcessTuple(HCPCSCodeLength, false));
    medicalClaimValidations.add(new ValidationProcessTuple(typeOfBillPresenseAndLengthForInstitutionalClaims, false));
    medicalClaimValidations.add(new ValidationProcessTuple(dischargeStatusPresenseAndLengthForInpatientClaims, false));
    medicalClaimValidations.add(new ValidationProcessTuple(diagnosisCodeFormat, false));