Question

1）我有一个需要处理的大文件（> 100k行）。我有很多业务验证和针对每个订单项的外部系统检查。代码正在从遗留应用程序迁移，我只是将这些业务逻辑放入AsyncitemProcessor中，AsyncitemProcessor也将数据保存到数据库中。这是在ItemProcessor中创建/保存记录（代替ItemWriter）的好习惯吗？ 2）代码是::

@Configuration
@EnableAutoConfiguration
@ComponentScan(basePackages = "com.liquidation.lpid")
@EntityScan(basePackages = "com.liquidation.lpid.entities")
@EnableTransactionManagement
public class SimpleJobConfiguration {
    @Autowired
    public JobRepository jobRepository;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    @Qualifier("myFtpSessionFactory")
    private SessionFactory myFtpSessionFactory;

    @Autowired
    public JobBuilderFactory jobBuilderFactory;


    @Bean
    public ThreadPoolTaskExecutor lpidItemTaskExecutor() {
        ThreadPoolTaskExecutor tExec = new ThreadPoolTaskExecutor();
        tExec.setCorePoolSize(10);
        tExec.setMaxPoolSize(10);
        tExec.setAllowCoreThreadTimeOut(true);
        return tExec;
    }

    @BeforeStep
    public void beforeStep(StepExecution stepExecution){
        String name = stepExecution.getStepName();
        System.out.println("name: " + name);
    }

    @Bean
    public SomeItemWriterListener someItemWriterListener(){
        return new SomeItemWriterListener();
    };

    @Bean
    @StepScope
    public FlatFileItemReader<FieldSet> lpidItemReader(@Value("#{stepExecutionContext['fileResource']}") String fileResource) {
        System.out.println("itemReader called !!!!!!!!!!! for customer data" + fileResource);

        FlatFileItemReader<FieldSet> reader = new FlatFileItemReader<FieldSet>();
        reader.setResource(new ClassPathResource("/data/stage/"+ fileResource));
        reader.setLinesToSkip(1);
        DefaultLineMapper<FieldSet> lineMapper = new DefaultLineMapper<FieldSet>();
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        reader.setSkippedLinesCallback(new LineCallbackHandler() {
            public void handleLine(String line) {
                if (line != null) {
                    tokenizer.setNames(line.split(","));
                }
            }
        });
        lineMapper.setLineTokenizer(tokenizer);
        lineMapper.setFieldSetMapper(new PassThroughFieldSetMapper());

        lineMapper.afterPropertiesSet();
        reader.setLineMapper(lineMapper);
        return reader;
    }


    @Bean
    public ItemWriter<FieldSet> lpidItemWriter() {
        return new LpidItemWriter();

    }

    @Autowired
    private  MultiFileResourcePartitioner multiFileResourcePartitioner;


    @Bean
    public Step masterStep() {
        return stepBuilderFactory.get("masterStep")
                .partitioner(slaveStep().getName(), multiFileResourcePartitioner)
                .step(slaveStep())
                .gridSize(4)
                .taskExecutor(lpidItemTaskExecutor())
                .build();   
    }

    @Bean
    public ItemProcessListener<FieldSet,String> processListener(){
        return new LpidItemProcessListener();
    }


    @SuppressWarnings("unchecked")
    @Bean
    public Step slaveStep() {
        return stepBuilderFactory.get("slaveStep")
                .<FieldSet,FieldSet>chunk(5)
                .faultTolerant()
                .listener(new ChunkListener())
                .reader(lpidItemReader(null))
                .processor(asyncItemProcessor())
         .writer(asyncItemWriter()).listener(someItemWriterListener()).build();
    }

    @Bean
    public AsyncItemWriter<FieldSet> asyncItemWriter(){
        AsyncItemWriter<FieldSet> asyncItemProcessor = new AsyncItemWriter<>();
        asyncItemProcessor.setDelegate(lpidItemWriter());
        try {
            asyncItemProcessor.afterPropertiesSet();
        } catch (Exception e) {
            e.printStackTrace();
        }

        return asyncItemProcessor;
    }

    @Bean
    public ItemProcessor<FieldSet, FieldSet> processor() {
        return new lpidCheckItemProcessor();
    }


    @Bean
    public AsyncItemProcessor<FieldSet, FieldSet> asyncItemProcessor() {
        AsyncItemProcessor<FieldSet, FieldSet> asyncItemProcessor = new AsyncItemProcessor<FieldSet, FieldSet>();

        asyncItemProcessor.setDelegate(processor());
        asyncItemProcessor.setTaskExecutor(lpidItemTaskExecutor());
        try {
            asyncItemProcessor.afterPropertiesSet();
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        return asyncItemProcessor;
    }


    @Bean
    public Job job() throws Exception {
        return jobBuilderFactory.get("job").incrementer(new RunIdIncrementer()).start(masterStep()).build();
    }
}

itemwriter在itemprocessor完成之前运行。我的理解是：对于每个块，项目读取器读取数据，项目处理器将在每个项目中流失，并且在块的末尾，项目编写器被调用（在我的情况下，它自迭代处理器以来不执行任何操作坚持数据）。但是在项目处理器完成之前，项目编写者会被调用，而我的工作永远不会完成。我在这里做错了什么？（我查看了之前的问题，解决方案是将编写器包装在AsyncItemWriter（）中，我正在这样做）。

由于 Sundar

Spring Batch：AsyncItemProcessor和AsyncItemWriter

0 个答案: