Question

我尝试在Windows中使用时间戳记模式创建源Kafka连接。我有这张桌子：

name=jdbc-conector  
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector  
connection.url=jdbc:oracle:thin:@localhost:1521:xe  
connection.user=usu  
connection.password=pwd  
query=select NDZ, to_timestamp(STRING_TIMESTAMP,'YYYY - MM - DD HH: MI: SS, FF6') AS TIMESTAMP_COLUMN FROM myTable
mode=timestamp+incrementing 
timestamp.column.name=TIMESTAMP_COLUMN
incrementing.column.name=NDZ
db.timezone=UTC
dialect.name=OracleDatabaseDialect
numeric.mapping=best_fit
table.types=TABLE  
poll.interval.ms=1000  
topic.prefix=input-mytable

INSERT INTO "usu"."mytable" (first_name,last_name) values("jake","tyler");
select output:
jake    tyler   2019-08-23 11:54:47:046

oracle数据库属性：

kafka-console-consumer --bootstrap-server localhost:9092 --topic input-mytable

控制台用户不返回任何输出：

curl localhost:8083/connectors/jdbc-conector/status/  
{"name":"jdbc-conector","connector": 
{"state":"RUNNING","worker_id":"1.2.3.4:8083"},"tasks": 
[{"id":0,"state":"RUNNING","worker_id":"1.2.3.4:8083"}],"type":"source"}

连接器仅在批量模式下工作。我不懂为什么。

检查连接器：

D:\kafka\bin\windows\connect-standalone.bat D:\kafka\config\connect-standalone.properties D:\kafka\config\connect-bbdd.properties

Ed ：在调试模式下，我的输出是这样的。

kafka连接源：

DEBUG: prepared SQL query: 'select NDZ, to_timestamp(STRING_TIMESTAMP,'YYYY - MM - DD HH: MI: SS, FF6') AS TIMESTAMP_COLUMN FROM myTable) WHERE "TIMESTAMP_COLUMN" < ? AND(("TIMESTAMP_COLUMN" = ? AND "NDZ" > ? ) OR "TIMESTAMP_COLUMN" > ? ) ORDER BY "TIMESTAMP_COLUMN", "NDZ" ASC.  

[2019-08-23 14:01:36,150] DEBUG Executing prepared statement with start time value = 2019-08-23 11:40:47:046 end time = 2019-08-23 11:40:47:046 and incrementing value = 19 (io.confluent.connect.jdbc.source.TimestampIncrementingCriteria)

输出：

select tk, tk2, count(*), count(a2.tk) / count(*) as ratio
from ((select distinct a.TK, a2.TK as tk2
       from access3 as a inner join
            access3 as a2 
            on a.DocumentNo = a2.DocumentNo
       where a.TK <> a2.TK
      ) tt left join
      access3 as a
      on a.tk = tt.tk
     ) left join
     access3 as a2
     on a2.tk = tt.tk2 and a2.documentNo = a.documentNo
group by tk, tk2;

我如何控制时差？

Answer 1

对我来说，这个问题与我的kafka连接器的时区有关。我的Kafka连接器是GMT time，而DB是GMT + 3，所以所有的inserts \ updates都提前了，因此连接器无法看到他们。

源kafka连接未使用时间戳记模式发布

1 个答案: