Question

几天前我发布了一个问题 - Flink Jdbc sink

现在，我正在尝试使用 flink 提供的接收器。

我已经编写了代码并且它也能正常工作。但是什么都没有保存在数据库中，也没有例外。使用以前的接收器我的代码没有完成（理想情况下应该作为它的流媒体应用程序发生）但是在下面的代码之后我没有收到错误并且没有任何东西被保存到数据库中。

public class CompetitorPipeline implements Pipeline {
    private final StreamExecutionEnvironment streamEnv;
    private final ParameterTool parameter;
   
    private static final Logger LOG = LoggerFactory.getLogger(CompetitorPipeline.class);

    public CompetitorPipeline(StreamExecutionEnvironment streamEnv, ParameterTool parameter) {
        this.streamEnv = streamEnv;
        this.parameter = parameter;
    }

    @Override
    public KeyedStream<CompetitorConfig, String> start(ParameterTool parameter) throws Exception {
        CompetitorConfigChanges competitorConfigChanges = new CompetitorConfigChanges();
        KeyedStream<CompetitorConfig, String> competitorChangesStream = competitorConfigChanges.run(streamEnv, parameter);
        

        //Add to JBDC Sink
        competitorChangesStream.addSink(JdbcSink.sink(
                "insert into competitor_config_universe(marketplace_id,merchant_id, competitor_name, comp_gl_product_group_desc," +
                        "category_code, competitor_type, namespace, qualifier, matching_type," +
                        "zip_region, zip_code, competitor_state, version_time, compConfigTombstoned, last_updated) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
                (ps, t) -> {
                    ps.setInt(1, t.getMarketplaceId());
                    ps.setLong(2, t.getMerchantId());
                    ps.setString(3, t.getCompetitorName());
                    ps.setString(4, t.getCompGlProductGroupDesc());
                    ps.setString(5, t.getCategoryCode());
                    ps.setString(6, t.getCompetitorType());
                    ps.setString(7, t.getNamespace());
                    ps.setString(8, t.getQualifier());
                    ps.setString(9, t.getMatchingType());
                    ps.setString(10, t.getZipRegion());
                    ps.setString(11, t.getZipCode());
                    ps.setString(12, t.getCompetitorState());
                    ps.setTimestamp(13, Timestamp.valueOf(t.getVersionTime()));
                    ps.setBoolean(14, t.isCompConfigTombstoned());
                    ps.setTimestamp(15, new Timestamp(System.currentTimeMillis()));
                    System.out.println("sql"+ps);
                },
                new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
                        .withUrl("jdbc:mysql://127.0.0.1:3306/database")
                        .withDriverName("com.mysql.cj.jdbc.Driver")
                        .withUsername("xyz")
                        .withPassword("xyz@")
                        .build()));
        return competitorChangesStream;
    }
}

Answer 1

您需要为 jdbc Sink 启用自动提交模式。

new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
                        .withUrl("jdbc:mysql://127.0.0.1:3306/database;autocommit=true")

看起来 SimpleBatchStatementExecutor 只能在自动提交模式下工作。而如果你需要提交和回滚批处理，那么你必须自己编写** JdbcBatchStatementExecutor **

Answer 2

您是否尝试包含 JdbcExecutionOptions ？

dataStream.addSink(JdbcSink.sink(
            sql_statement,
            (statement, value) -> {
                /* Prepared Statement */
            }, 
            JdbcExecutionOptions.builder()
                    .withBatchSize(5000)
                    .withBatchIntervalMs(200)
                    .withMaxRetries(2)
                    .build(),
            new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
                    .withUrl("jdbc:mysql://127.0.0.1:3306/database")
                    .withDriverName("com.mysql.cj.jdbc.Driver")
                    .withUsername("xyz")
                    .withPassword("xyz@")
                    .build()));

Flink JDBC 接收器第 2 部分

2 个答案: