此问题基于KSQL stream from topic with heterogeneous JSON structures中的解决方案路径-尝试从具有select
的主题创建的流中尝试value_format='json'
时遇到错误。
我仍然可以print
作为基础主题,但是执行select * from stream;
会引发错误。
我们正在使用Confluent 5.3.1,又名Kafka 2.3.1。
这是我(重新)创建问题的方式...
ksql> PRINT 'sms-reporting.status-updated';
Format:JSON
{"ROWTIME":1573688864403,"ROWKEY":"null","acceptTs":"1573603201000","eventType":"REPORTING","isFinal":"true","messageId":"8619Z-1113H-00007-01CF5","nodeId":"86","responseCode":"3041","submissionAttempt":"1","ts":"1573603201567","type":"statusUpdated"}
ksql> CREATE STREAM statusupdated ( \
acceptTs VARCHAR, \
eventType VARCHAR, \
isFinal VARCHAR, \
messageId VARCHAR, \
nodeId VARCHAR, \
responseCode VARCHAR, \
submissionAttempt VARCHAR, \
ts VARCHAR, \
type VARCHAR \
) with (kafka_topic='sms-reporting.status-updated',value_format='json');
Message
----------------
Stream created
----------------
ksql> DESCRIBE statusupdated;
Name : STATUSUPDATED
Field | Type
-----------------------------------------------
ROWTIME | BIGINT (system)
ROWKEY | VARCHAR(STRING) (system)
ACCEPTTS | VARCHAR(STRING)
EVENTTYPE | VARCHAR(STRING)
ISFINAL | VARCHAR(STRING)
MESSAGEID | VARCHAR(STRING)
NODEID | VARCHAR(STRING)
RESPONSECODE | VARCHAR(STRING)
SUBMISSIONATTEMPT | VARCHAR(STRING)
TS | VARCHAR(STRING)
TYPE | VARCHAR(STRING)
-----------------------------------------------
ksql> SELECT * FROM statusupdated;
Query terminated
java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000","REPORTING","true","8619Z-1113H-00007-01CF5","86","3041","1","1573603201567","statusUpdated"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 206] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000","REPORTING","true","8619Z-1113H-00007-01CF5","86","3041","1","1573603201567","statusUpdated"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 206] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000","REPORTING","true","8619Z-1113H-00007-01CF5","86","3041","1","1573603201567","statusUpdated"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 206] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
如果我现在使用此流(在该流上我仍然无法执行select
语句而没有收到错误)创建另一个流(当然,该流将由一个 new 主题),我仍然无法select
观看新视频,但是我可以 print
支持主题,数据将按预期显示。
ksql> CREATE STREAM statusupdated2 AS SELECT acceptts FROM statusupdated;
Message
----------------------------
Stream created and running
----------------------------
ksql> describe statusupdated2;
Name : STATUSUPDATED2
Field | Type
--------------------------------------
ROWTIME | BIGINT (system)
ROWKEY | VARCHAR(STRING) (system)
ACCEPTTS | VARCHAR(STRING)
--------------------------------------
ksql> SELECT * FROM statusupdated2;
Query terminated
java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 113] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 113] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 113] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
print
新流的支持主题:ksql> PRINT statusupdated2;
Format:JSON
{"ROWTIME":1573692743435,"ROWKEY":"null","ACCEPTTS":"1573603201000"}
答案 0 :(得分:0)
您尝试过
SET'auto.offset.reset'='最早';
在运行选择查询之前?