我正在使用JdbcPagingItemReader进行Spring Batch中的分区从属步骤。
以下是我的JobConfiguration for Partitioning步骤
@Bean
public Job loadConsumerJob() {
return jobBuilderFactory.get(JOB_NAME)
.incrementer(new RunIdIncrementer())
.start(step1Master())
.build();
}
public Step step1Master() {
return stepBuilderFactory.get(STEP_ONE_MASTER_NAME)
.<ConsumerNoMasterInterface, BillInterface> partitioner(STEP_ONE_NAME,jobTwoStepOneMasterConfiguration.jobTwoStepOnePartioner(null))
.step(step1())
.gridSize(jobTwoStepOneMasterConfiguration.jobTwoStepOneGridSize())
.taskExecutor(jobTwoStepOneMasterConfiguration.jobTwoStepOneTaskExecutor())
.listener(jobTwoStepOneMasterConfiguration.jobTwoStepOneMasterListener())
.build();
}
public Step step1() {
return stepBuilderFactory.get(STEP_ONE_NAME)
.<ConsumerNoMasterInterface, BillInterface> chunk(CHUNK_SIZE)
.reader(jobTwoStepOneConfiguration.jobTwoStepOneJdbcPagingItemReader(null,null,0,0,0))
.processor(jobTwoStepOneConfiguration.jobTwoStepOneProcessor())
.writer(jobTwoStepOneConfiguration.jobTwoStepOneWriter())
.build();
}
这是我的QueryProvider
@Bean
public PostgresPagingQueryProvider jobTwoStepOneQueryProvider() {
String methodName = "jobTwoStepOneQueryProvider() : ";
logger.info(methodName + "called");
PostgresPagingQueryProvider provider = new PostgresPagingQueryProvider();
provider.setSelectClause("select * ");
provider.setFromClause("from (\n" +
"select cnm.*,row_number() over(order by id) as rownum from consumer_no_master cnm where \n" +
"group_no=:groupNo AND consumer_no not in (select consumer_no from bill b where b.bill_month=:billMonth and deleted = false)\n" +
") as consumers ");
provider.setWhereClause("where rownum between :fromRow AND :toRow ");
Map<String,Order> sortKeys = new HashMap<>();
sortKeys.put("rownum",Order.ASCENDING);
provider.setSortKeys(sortKeys);
return provider;
}
这是我的JdbcPagingItemReaderBuilder代码
@Bean
@StepScope
public JdbcPagingItemReader jobTwoStepOneJdbcPagingItemReader(@Value("#{jobParameters['GROUP_NO']}") String groupNo,
@Value("#{jobParameters['BILL_MONTH']}") String billMonth,
@Value("#{stepExecutionContext[fromRow]}") int fromRow,
@Value("#{stepExecutionContext[toRow]}") int toRow,
@Value("#{stepExecutionContext[limit]}") int limit) {
String methodName = "jobTwoStepOneJdbcPagingItemReader() : ";
logger.info(methodName + "called for groupNo " + groupNo + " billMonth " + billMonth + " fromRow " + fromRow + " toRow " + toRow + " with limit " + limit);
Map<String, Object> parameterValues = new HashMap<>();
parameterValues.put("groupNo", groupNo);
parameterValues.put("billMonth",billMonth);
parameterValues.put("fromRow",fromRow);
parameterValues.put("toRow",toRow);
return new JdbcPagingItemReaderBuilder<ConsumerNoMaster>()
.name("consumerReader")
.dataSource(dataSource)
.queryProvider(jobTwoStepOneQueryProvider())
.parameterValues(parameterValues)
.rowMapper(new ConsumerNoMasterRowMapper())
.pageSize(limit)
.build();
}
这是我的分区功能
@Override
public Map<String, ExecutionContext> partition(int gridSize) {
final String methodName = "partition() : ";
logger.info(methodName + "called with gridSize " + gridSize + " chunk size " + chunkSize + " total size" + totalSize + " for group " + groupNo);
Map<String, ExecutionContext> partitions = new HashMap<String, ExecutionContext>();
if(gridSize <= 0 || totalSize <= 0 || chunkSize <= 0){
logger.error(methodName + "Either of Param is <= 0");
return partitions;
}
int work = new BigDecimal(totalSize).divide(new BigDecimal(gridSize),0,RoundingMode.CEILING).intValue();
logger.info(methodName + "for group " + groupNo + " with total size" + totalSize + " and grid size " + gridSize + " unit of work for each cpu " + work);
int fromRow = 1;
int toRow = work;
for(int i = 1; i <= gridSize; i++){
logger.info(methodName + "For group " + groupNo + " with partition-" + i + " fromRow " + fromRow + " toRow " + toRow);
ExecutionContext executionContext = new ExecutionContext();
executionContext.put(JobTwoStepOneConfiguration.FROM_ROW,fromRow);
executionContext.put(JobTwoStepOneConfiguration.TO_ROW,toRow);
executionContext.put(JobTwoStepOneConfiguration.LIMIT,work);
partitions.put("partition-" + i,executionContext);
fromRow = toRow + 1;
toRow = toRow + work;
}
return partitions;
}
当我使用1157的消费者大小运行我的工作时,JdbcPagingItemReader触发的SQL就是这个(从日志中捕获)
2018-06-15 13:56:41.728 DEBUG 14116 --- [SimpleAsyncTaskExecutor-4] o.s.b.i.database.JdbcPagingItemReader : SQL used for reading remaining pages: [SELECT * FROM (
select cnm.*,row_number() over(order by id) as rownum from consumer_no_master cnm where
group_no=:groupNo AND consumer_no not in (select consumer_no from bill b where b.bill_month=:billMonth and deleted = false)
) as consumers WHERE (rownum between :fromRow AND :toRow) AND ((rownum > :_rownum)) ORDER BY rownum ASC LIMIT 290]
2018-06-15 13:56:41.728 DEBUG 14116 --- [SimpleAsyncTaskExecutor-4] o.s.b.i.database.JdbcPagingItemReader : Using parameterMap:{billMonth=MAY-2018, fromRow=1, toRow=290, groupNo=KSK13, _rownum=290}
我不知道为什么JdbcPagingItemReader在生成的SQL语句中设置_rownum。我没有将它设置为任何参数值映射。
我想要的是在传递fromRow和toRow值之间读取消费者为什么JdbcPagingItemReader在SQL中附加_rownum参数???
请指导我在这里做错了!!!