KSQL:尝试从流中选择*

时间:2019-11-14 01:02:03

标签: ksql

此问题基于KSQL stream from topic with heterogeneous JSON structures中的解决方案路径-尝试从具有select的主题创建的流中尝试value_format='json'时遇到错误。  我仍然可以print作为基础主题,但是执行select * from stream;会引发错误。

我们正在使用Confluent 5.3.1,又名Kafka 2.3.1。

这是我(重新)创建问题的方式...

  1. 从一些简单的json数据开始
ksql> PRINT 'sms-reporting.status-updated';
Format:JSON
{"ROWTIME":1573688864403,"ROWKEY":"null","acceptTs":"1573603201000","eventType":"REPORTING","isFinal":"true","messageId":"8619Z-1113H-00007-01CF5","nodeId":"86","responseCode":"3041","submissionAttempt":"1","ts":"1573603201567","type":"statusUpdated"}
  1. 创建流:
ksql> CREATE STREAM statusupdated ( \
 acceptTs VARCHAR, \
 eventType VARCHAR, \
 isFinal VARCHAR, \
 messageId VARCHAR, \
 nodeId VARCHAR, \
 responseCode VARCHAR, \
 submissionAttempt VARCHAR, \
 ts VARCHAR, \
 type VARCHAR \
) with (kafka_topic='sms-reporting.status-updated',value_format='json');

 Message
----------------
 Stream created
----------------
ksql> DESCRIBE statusupdated;

Name                 : STATUSUPDATED
 Field             | Type
-----------------------------------------------
 ROWTIME           | BIGINT           (system)
 ROWKEY            | VARCHAR(STRING)  (system)
 ACCEPTTS          | VARCHAR(STRING)
 EVENTTYPE         | VARCHAR(STRING)
 ISFINAL           | VARCHAR(STRING)
 MESSAGEID         | VARCHAR(STRING)
 NODEID            | VARCHAR(STRING)
 RESPONSECODE      | VARCHAR(STRING)
 SUBMISSIONATTEMPT | VARCHAR(STRING)
 TS                | VARCHAR(STRING)
 TYPE              | VARCHAR(STRING)
-----------------------------------------------
  1. 尝试(失败)查看流的内容:
ksql> SELECT * FROM statusupdated;
Query terminated
java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
 at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000","REPORTING","true","8619Z-1113H-00007-01CF5","86","3041","1","1573603201567","statusUpdated"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 206] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
 at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000","REPORTING","true","8619Z-1113H-00007-01CF5","86","3041","1","1573603201567","statusUpdated"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 206] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
 at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000","REPORTING","true","8619Z-1113H-00007-01CF5","86","3041","1","1573603201567","statusUpdated"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 206] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])

如果我现在使用此流(在该流上我仍然无法执行select语句而没有收到错误)创建另一个流(当然,该流将由一个 new 主题),我仍然无法select观看新视频,但是可以 print支持主题,数据将按预期显示。

  1. 创建派生流
ksql> CREATE STREAM statusupdated2 AS SELECT acceptts FROM statusupdated;

 Message
----------------------------
 Stream created and running
----------------------------
ksql> describe statusupdated2;

Name                 : STATUSUPDATED2
 Field    | Type
--------------------------------------
 ROWTIME  | BIGINT           (system)
 ROWKEY   | VARCHAR(STRING)  (system)
 ACCEPTTS | VARCHAR(STRING)
--------------------------------------
  1. 尝试(失败)查看派生流的内容:
ksql> SELECT * FROM statusupdated2;
Query terminated
java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
 at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 113] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
 at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 113] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
Caused by: Unrecognized field "terminal" (class io.confluent.ksql.rest.entity.StreamedRow), not marked as ignorable (3 known properties: "finalMessage", "row", "errorMessage"])
 at [Source: (String)"{"row":{"columns":[1573692743435,null,"1573603201000"]},"errorMessage":null,"finalMessage":null,"terminal":false}"; line: 1, column: 113] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["terminal"])
  1. print新流的支持主题:
ksql> PRINT statusupdated2;
Format:JSON
{"ROWTIME":1573692743435,"ROWKEY":"null","ACCEPTTS":"1573603201000"}

1 个答案:

答案 0 :(得分:0)

您尝试过

  

SET'auto.offset.reset'='最早';

在运行选择查询之前?