spring批量ItemWriter创建输出两次

时间:2016-07-22 03:11:02

标签: spring spring-batch

我编写了一个弹出批处理作业,它从数据库读取然后写入csv。

这项工作非常简单,但是无论ItemWriter看起来像是执行了两次,它都能正常工作。即我最终得到两个CSV(完全相同),而不是一个。

有人可以帮我解决原因吗?它使用不同的jobExecution.id。

附加以下运行的批处理作业和控制台输出...

@Configuration
@EnableBatchProcessing
public class ExportMasterListCsvJobConfig {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job exportMasterListCsvJob(
            Step readStgDbAndExportMasterListStep, 
            Step writeToCsvStep) {

        return jobBuilderFactory.get("exportMasterListCsvJob")
                .incrementer(new RunIdIncrementer())
                .flow(readStgDbAndExportMasterListStep)
                .end()
                .build();
    }
    @Bean
    public Step readStgDbAndExportMasterListStep(
            @Value("${exportMasterListCsv.generateMasterListRows.chunkSize}") int chunkSize,
            @Qualifier("queryOnlineStagingDbReader") ItemReader<MasterList> queryOnlineStagingDbReader,
            @Qualifier("masterListFileWriter") ItemWriter<MasterList> masterListFileWriter) {

        return  stepBuilderFactory.get("readStgDbAndExportMasterListStep")
                    .<MasterList,MasterList>chunk(chunkSize)
                    .reader(queryOnlineStagingDbReader)
                    .writer(masterListFileWriter)
                    .build();

    }   
    @Bean
    public ItemReader<MasterList> queryOnlineStagingDbReader(
            DataSource onlineStagingDb,
            @Value("${exportMasterListCsv.generateMasterListRows.masterListSql}") String masterListSql) {

        JdbcCursorItemReader<MasterList> reader = new JdbcCursorItemReader<>();

        reader.setDataSource(onlineStagingDb);
        reader.setSql(masterListSql);       
        reader.setRowMapper(new RowMapper<MasterList>() { 
            @Override 
            public MasterList mapRow(ResultSet resultSet, int i) throws SQLException {
                MasterList masterList = new MasterList();
                masterList.setL2(resultSet.getString("l2"));
                masterList.setL2Name(resultSet.getString("l2_name"));
                return masterList;
            } 
        });         
        return reader;
    }   
    @Bean
    @StepScope
    public ItemWriter<MasterList> masterListFileWriter(
            FileSystemResource masterListFile,
            @Value("#{stepExecutionContext}")Map<String, Object> executionContext) {

        FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
        writer.setResource(masterListFile);

        DelimitedLineAggregator<MasterList> lineAggregator = new DelimitedLineAggregator<>();
        lineAggregator.setDelimiter(",");

        BeanWrapperFieldExtractor<MasterList> extractor = new BeanWrapperFieldExtractor<MasterList>();
        extractor.setNames(new String[] { "l2", "l2Name"});
        lineAggregator.setFieldExtractor(extractor);

        writer.setLineAggregator(lineAggregator);
        writer.setForceSync(true);
        writer.open(new ExecutionContext(executionContext));
        return writer;
    }   
    @Bean
    @JobScope
    public FileSystemResource masterListFile(
            @Value("${temp.dir}") String tempDir,
            @Value("#{jobExecution.id}") Long jobId
            ) throws IOException {

        return new FileSystemResource(new File(tempDir, "masterList" + jobId + ".csv"));
    }
}

和控制台输出......

13:03:45.815 [main] DEBUG org.springframework.test.context.junit4.SpringJUnit4ClassRunner - SpringJUnit4ClassRunner constructor called with [class com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.827 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating CacheAwareContextLoaderDelegate from class [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
13:03:45.836 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating BootstrapContext using constructor [public org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
13:03:45.853 [main] DEBUG org.springframework.test.context.BootstrapUtils - Instantiating TestContextBootstrapper for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest] from class [org.springframework.test.context.support.DefaultTestContextBootstrapper]
13:03:45.884 [main] DEBUG org.springframework.test.context.support.DefaultTestContextBootstrapper - Found explicit ContextLoader class [org.springframework.boot.test.SpringApplicationContextLoader] for context configuration attributes [ContextConfigurationAttributes@3cbbc1e0 declaringClass = 'com.myer.pricing.onlinestore.export.ExportMasterListCsvTest', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', locations = '{}', inheritLocations = true, initializers = '{}', inheritInitializers = true, name = [null], contextLoaderClass = 'org.springframework.boot.test.SpringApplicationContextLoader']
13:03:45.899 [main] DEBUG org.springframework.test.context.support.AbstractContextLoader - Did not detect default resource location for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]: class path resource [com/myer/pricing/onlinestore/export/ExportMasterListCsvTest-context.xml] does not exist
13:03:45.900 [main] DEBUG org.springframework.test.context.support.AbstractContextLoader - Did not detect default resource location for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]: class path resource [com/myer/pricing/onlinestore/export/ExportMasterListCsvTestContext.groovy] does not exist
13:03:45.900 [main] INFO org.springframework.test.context.support.AbstractContextLoader - Could not detect default resource locations for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]: no resource found for suffixes {-context.xml, Context.groovy}.
13:03:45.927 [main] INFO org.springframework.test.context.support.DefaultTestContextBootstrapper - Using TestExecutionListeners: [org.springframework.boot.test.IntegrationTestPropertiesListener@724af044, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener@4678c730, org.springframework.test.context.support.DependencyInjectionTestExecutionListener@6767c1fc, org.springframework.test.context.support.DirtiesContextTestExecutionListener@29ee9faa, org.springframework.test.context.transaction.TransactionalTestExecutionListener@c038203, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener@cc285f4]
13:03:45.931 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved @ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.931 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.946 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved @ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.946 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.948 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved @ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.948 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.950 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved @ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.950 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.953 [main] DEBUG org.springframework.test.context.support.AbstractDirtiesContextTestExecutionListener - Before test class: context [DefaultTestContext@6a4f787b testClass = ExportMasterListCsvTest, testInstance = [null], testMethod = [null], testException = [null], mergedContextConfiguration = [MergedContextConfiguration@685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]], class annotated with @DirtiesContext [false] with mode [null].
13:03:45.954 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved @ProfileValueSourceConfiguration [null] for test class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.954 [main] DEBUG org.springframework.test.annotation.ProfileValueUtils - Retrieved ProfileValueSource type [class org.springframework.test.annotation.SystemProfileValueSource] for class [com.myer.pricing.onlinestore.export.ExportMasterListCsvTest]
13:03:45.959 [main] DEBUG org.springframework.test.util.ReflectionTestUtils - Getting field 'mergedContextConfiguration' from target object [[DefaultTestContext@6a4f787b testClass = ExportMasterListCsvTest, testInstance = com.myer.pricing.onlinestore.export.ExportMasterListCsvTest@53b32d7, testMethod = [null], testException = [null], mergedContextConfiguration = [MergedContextConfiguration@685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]]] or target class [class org.springframework.test.context.support.DefaultTestContext]
13:03:45.960 [main] DEBUG org.springframework.test.util.ReflectionTestUtils - Setting field 'propertySourceProperties' of type [null] on target object [[MergedContextConfiguration@685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]] or target class [class org.springframework.test.context.MergedContextConfiguration] to value [[Ljava.lang.String;@4667ae56]
13:03:45.961 [main] DEBUG org.springframework.test.context.support.DependencyInjectionTestExecutionListener - Performing dependency injection for test context [[DefaultTestContext@6a4f787b testClass = ExportMasterListCsvTest, testInstance = com.myer.pricing.onlinestore.export.ExportMasterListCsvTest@53b32d7, testMethod = [null], testException = [null], mergedContextConfiguration = [MergedContextConfiguration@685cb137 testClass = ExportMasterListCsvTest, locations = '{}', classes = '{class com.myer.pricing.onlinestore.export.TestJobConfig, class com.myer.pricing.onlinestore.export.job.ExportMasterListCsvJobConfig}', contextInitializerClasses = '[]', activeProfiles = '{batchtest}', propertySourceLocations = '{}', propertySourceProperties = '{org.springframework.boot.test.IntegrationTest=true}', contextLoader = 'org.springframework.boot.test.SpringApplicationContextLoader', parent = [null]]]].
13:03:46.072 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [systemProperties] PropertySource with lowest search precedence
13:03:46.073 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [systemEnvironment] PropertySource with lowest search precedence
13:03:46.074 [main] DEBUG org.springframework.core.env.StandardEnvironment - Initialized StandardEnvironment with PropertySources [systemProperties,systemEnvironment]
13:03:46.074 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [test] PropertySource with highest search precedence
13:03:46.075 [main] DEBUG org.springframework.core.env.StandardEnvironment - Adding [integrationTest] PropertySource with search precedence immediately lower than [systemEnvironment]

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.3.6.RELEASE)

2016-07-22 13:03:46.541  INFO 10084 --- [           main] c.m.p.o.export.ExportMasterListCsvTest   : Starting ExportMasterListCsvTest on IGSM473537 with PID 10084 (C:\design_centre_dev\myer-pricing-engine\java-onlinestore-feed-exporter\target\test-classes started by rriviere in C:\design_centre_dev\myer-pricing-engine\java-onlinestore-feed-exporter)
2016-07-22 13:03:46.542  INFO 10084 --- [           main] c.m.p.o.export.ExportMasterListCsvTest   : The following profiles are active: batchtest
2016-07-22 13:03:46.875  INFO 10084 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@4e08711f: startup date [Fri Jul 22 13:03:46 AEST 2016]; root of context hierarchy
2016-07-22 13:03:48.151  INFO 10084 --- [           main] o.s.b.f.config.PropertiesFactoryBean     : Loading properties file from URL [jar:file:/C:/.m2/repository/org/springframework/integration/spring-integration-core/4.2.8.RELEASE/spring-integration-core-4.2.8.RELEASE.jar!/META-INF/spring.integration.default.properties]
2016-07-22 13:03:48.157  INFO 10084 --- [           main] o.s.i.config.IntegrationRegistrar        : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2016-07-22 13:03:48.716  WARN 10084 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.stepScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean javadoc for complete details.
2016-07-22 13:03:48.728  WARN 10084 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.jobScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean javadoc for complete details.
2016-07-22 13:03:48.742  INFO 10084 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2016-07-22 13:03:48.751  INFO 10084 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2016-07-22 13:03:48.900  INFO 10084 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$402d705e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-07-22 13:03:49.060  INFO 10084 --- [           main] o.s.j.d.e.EmbeddedDatabaseFactory        : Starting embedded database: url='jdbc:hsqldb:mem:testdb', username='sa'
2016-07-22 13:03:49.459  INFO 10084 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [db/sql/staging_db.sql]
2016-07-22 13:03:49.465  INFO 10084 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [db/sql/staging_db.sql] in 6 ms.
2016-07-22 13:03:49.710  WARN 10084 --- [           main] o.s.b.c.l.AbstractListenerFactoryBean    : org.springframework.batch.item.ItemWriter is an interface.  The implementing class will not be queried for annotation based listener configurations.  If using @StepScope on a @Bean method, be sure to return the implementing class so listner annotations can be used.
2016-07-22 13:03:49.797  INFO 10084 --- [           main] o.s.aop.framework.CglibAopProxy          : Unable to proxy method [public final java.lang.String org.springframework.core.io.FileSystemResource.getPath()] because it is final: All calls to this method via a proxy will NOT be routed to the target instance.
2016-07-22 13:03:50.236  INFO 10084 --- [           main] o.s.b.f.config.PropertiesFactoryBean     : Loading properties file from URL [jar:file:/C:/.m2/repository/org/springframework/integration/spring-integration-core/4.2.8.RELEASE/spring-integration-core-4.2.8.RELEASE.jar!/META-INF/spring.integration.default.properties]
2016-07-22 13:03:50.730  INFO 10084 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService  'taskScheduler'
2016-07-22 13:03:51.258  INFO 10084 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2016-07-22 13:03:51.281  INFO 10084 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 23 ms.
2016-07-22 13:03:51.813  INFO 10084 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
2016-07-22 13:03:51.814  INFO 10084 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2016-07-22 13:03:51.814  INFO 10084 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'application:batchtest.errorChannel' has 1 subscriber(s).
2016-07-22 13:03:51.814  INFO 10084 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started _org.springframework.integration.errorLogger
2016-07-22 13:03:51.832  INFO 10084 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
2016-07-22 13:03:51.844  INFO 10084 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: HSQL
2016-07-22 13:03:51.973  INFO 10084 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2016-07-22 13:03:52.044  INFO 10084 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=exportMasterListCsvJob]] launched with the following parameters: [{run.id=1}]
2016-07-22 13:03:52.073  INFO 10084 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [readStgDbAndExportMasterListStep]
2016-07-22 13:03:52.167  INFO 10084 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=exportMasterListCsvJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
2016-07-22 13:03:52.169  INFO 10084 --- [           main] c.m.p.o.export.ExportMasterListCsvTest   : Started ExportMasterListCsvTest in 6.091 seconds (JVM running for 6.895)
2016-07-22 13:03:52.196  INFO 10084 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=exportMasterListCsvJob]] launched with the following parameters: [{random=93853}]
2016-07-22 13:03:52.234  INFO 10084 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [readStgDbAndExportMasterListStep]
2016-07-22 13:03:52.290  INFO 10084 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=exportMasterListCsvJob]] completed with the following parameters: [{random=93853}] and the following status: [COMPLETED]
2016-07-22 13:03:52.305  INFO 10084 --- [       Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@4e08711f: startup date [Fri Jul 22 13:03:46 AEST 2016]; root of context hierarchy
2016-07-22 13:03:52.307  INFO 10084 --- [       Thread-1] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2016-07-22 13:03:52.308  INFO 10084 --- [       Thread-1] o.s.i.endpoint.EventDrivenConsumer       : Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2016-07-22 13:03:52.309  INFO 10084 --- [       Thread-1] o.s.i.channel.PublishSubscribeChannel    : Channel 'application:batchtest.errorChannel' has 0 subscriber(s).
2016-07-22 13:03:52.309  INFO 10084 --- [       Thread-1] o.s.i.endpoint.EventDrivenConsumer       : stopped _org.springframework.integration.errorLogger
2016-07-22 13:03:52.314  INFO 10084 --- [       Thread-1] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'taskScheduler'
2016-07-22 13:03:52.317  INFO 10084 --- [       Thread-1] o.s.j.d.e.EmbeddedDatabaseFactory        : Shutting down embedded database: url='jdbc:hsqldb:mem:testdb'

这会生成两个CSV,每个CSV都有一个不同的jobExecution.id,我已将其附加到文件名中。 masterList0.csv和masterList1.csv

提前致谢

1 个答案:

答案 0 :(得分:2)

我的yml文件中有以下设置......

弹簧:   批量:     job.enabled:true

这导致我的工作运行两次。将其更改为false后,它解决了我的问题。