Question

我正在尝试使用以kafka作为代理的spring cloud数据流从oracle DB捕获更改数据。我为此使用轮询机制。我定期使用基本的选择查询轮询数据库，以捕获所有更新的数据。为了获得更好的防故障系统，我将上次轮询时间保留在oracle DB中，并用它来获取上次轮询后更新的数据。

public MessageSource<Object> jdbcMessageSource() {
    JdbcPollingChannelAdapter jdbcPollingChannelAdapter =
            new JdbcPollingChannelAdapter(this.dataSource, this.properties.getQuery());
    jdbcPollingChannelAdapter.setUpdateSql(this.properties.getUpdate());
    return jdbcPollingChannelAdapter;
}

@Bean
public IntegrationFlow pollingFlow() {
    IntegrationFlowBuilder flowBuilder = IntegrationFlows.from(jdbcMessageSource(),spec -> spec.poller(Pollers.fixedDelay(3000)));
    flowBuilder.channel(this.source.output());
    flowBuilder.transform(trans,"transform");
    return flowBuilder.get();

}

我在应用程序属性中的查询如下：

query: select * from kafka_test where LAST_UPDATE_TIME >(select LAST_POLL_TIME from poll_time)

update : UPDATE poll_time SET LAST_POLL_TIME = CURRENT_TIMESTAMP

这对我来说非常有效。我可以使用这种方法从数据库获取CDC。

我现在正在寻找的问题如下：

创建一个仅用于维护轮询时间的表是一个负担。我希望在kafka主题中保留最后一次轮询时间，并在进行下一次轮询时从kafka主题中检索该时间。

我修改了jdbcMessageSource方法，如下所示：

public MessageSource<Object> jdbcMessageSource() {
    String query = "select * from kafka_test where LAST_UPDATE_TIME > '"+<Last poll time value read from kafka comes here>+"'";

    JdbcPollingChannelAdapter jdbcPollingChannelAdapter =
            new JdbcPollingChannelAdapter(this.dataSource, query);
    return jdbcPollingChannelAdapter;
}

但是Spring数据流仅实例化pollingFlow（）（请参见上面的代码）bean。因此，首先运行的查询将保持不变。我想使用每次轮询的新轮询时间来更新查询。

有没有一种方法可以编写自定义Integrationflow来使每次查询时更新此查询？

我为此尝试了IntegrationFlowContext，但没有成功。

提前谢谢！！！

Answer 1

有关标准适配器中动态查询的机制，请参见Artem的答案；但是，另一种方法是将JdbcTemplate封装在Bean中，然后用

调用

IntegrationFlows.from(myPojo(), "runQuery", e -> ...)
    ...

甚至是一个简单的lambda

    .from(() -> jdbcTemplate...)

Answer 2

我们有此测试配置（对不起，它是XML）：

<inbound-channel-adapter query="select * from item where status=:status" channel="target"
                             data-source="dataSource" select-sql-parameter-source="parameterSource"
                             update="delete from item"/>


    <beans:bean id="parameterSource" factory-bean="parameterSourceFactory"
                factory-method="createParameterSourceNoCache">
        <beans:constructor-arg value=""/>
    </beans:bean>

    <beans:bean id="parameterSourceFactory"
                class="org.springframework.integration.jdbc.ExpressionEvaluatingSqlParameterSourceFactory">
        <beans:property name="parameterExpressions">
            <beans:map>
                <beans:entry key="status" value="@statusBean.which()"/>
            </beans:map>
        </beans:property>
        <beans:property name="sqlParameterTypes">
            <beans:map>
                <beans:entry key="status" value="#{ T(java.sql.Types).INTEGER}"/>
            </beans:map>
        </beans:property>
    </beans:bean>

    <beans:bean id="statusBean"
                class="org.springframework.integration.jdbc.config.JdbcPollingChannelAdapterParserTests$Status"/>

请注意ExpressionEvaluatingSqlParameterSourceFactory及其createParameterSourceNoCache()工厂。此结果可用于select-sql-parameter-source。

JdbcPollingChannelAdapter对此事有setSelectSqlParameterSource。

因此，您配置ExpressionEvaluatingSqlParameterSourceFactory以便能够将某些查询参数解析为某些bean方法调用的表达式，以便从Kafka获得所需的值。然后createParameterSourceNoCache()将帮助您获得预期的SqlParameterSource。

文档中也有一些信息：https://docs.spring.io/spring-integration/docs/current/reference/html/#jdbc-inbound-channel-adapter

Answer 3

With the help of both the answer above, I was able to figure out the approach. Write a jdbc template and wrap that as a bean and use it for the Integration Flow.

@EnableBinding(Source.class)
@AllArgsConstructor
public class StockSource {

  private DataSource dataSource;

  @Autowired
  private JdbcTemplate jdbcTemplate;

  private MessageChannelFactory messageChannelFactory;  // You can use normal message channel which is available in spring cloud data flow as well.

  private List<String> findAll() {
    jdbcTemplate = new JdbcTemplate(dataSource);
    String time = "10/24/60" . (this means 10 seconds for oracle DB)
    String query = << your query here like.. select * from test where (last_updated_time > time) >>;
    return jdbcTemplate.query(query, new RowMapper<String>() {
      @Override
      public String mapRow(ResultSet rs, int rowNum) throws SQLException {
          ...
          ...
          any row mapper operations that you want to do with you result after the poll.
          ...
          ...
          ...
        // Change the time here for the next poll to the DB. 
        return result;
      }
    });
  }

  @Bean
  public IntegrationFlow supplyPollingFlow() {

    IntegrationFlowBuilder flowBuilder = IntegrationFlows
        .from(this::findAll, spec -> {
          spec.poller(Pollers.fixedDelay(5000));
        });
    flowBuilder.channel(<<Your message channel>>);
    return flowBuilder.get();
  }

}

In our use case, we were persisting the last poll time in a kafka topic. This was to make the application state less. Every new poll to the DB now, will have a new time in the where condition.

P.S: your messaging broker (kafka/rabbit mq) sdould be running in your local or connect to them if there are hosted on a different platform.

God Speed !!!

具有Jdbc的Spring Integration Flow具有动态查询的消息源

3 个答案: