Question

我正在尝试基于sourceTopic1和sourceTopic2在targetTopic1中创建数据。两个源主题都应该具有相同的事件结构。首先创建目标流，然后尝试将数据从另一个源流插入当前流。有什么建议吗？

ksql> CREATE STREAM sourceTopic1Stream (category varchar, source varchar, type varchar, id varchar, payload varchar) WITH (KAFKA_TOPIC='sourceTopic1', VALUE_FORMAT='json');

 Message
----------------
 Stream created
----------------
ksql> CREATE STREAM sourceTopic2Stream (category varchar, source varchar, type varchar, id varchar, payload varchar) WITH (KAFKA_TOPIC='sourceTopic2', VALUE_FORMAT='json');

 Message
----------------
 Stream created
----------------
ksql> CREATE STREAM targetTopic1Stream WITH (kafka_topic='targetTopic1', partitions=3) AS select 'sourceTopic1' topicname, category, source, type, id, payload from sourceTopic1Stream where id like 'myid%';

 Message
----------------------------
 Stream created and running
----------------------------
ksql> INSERT INTO targetTopic1Stream SELECT 'sourceTopic2' topicname, category, source, type, id, payload FROM sourceTopic2Stream where id like 'myid%';
io.confluent.ksql.util.KsqlException: Sink topic TARGETTOPIC1STREAM does not exist in th e metastore.

ksql> show topics;

 Kafka Topic        | Registered | Partitions | Partition Replicas | Consumers | ConsumerGroups
------------------------------------------------------------------------------------------------
 _confluent-metrics | false      | 12         | 1                  | 0         | 0
 _schemas           | false      | 1          | 1                  | 0         | 0
 sourceTopic1       | true       | 3          | 1                  | 3         | 1
 sourceTopic2       | true       | 3          | 1                  | 0         | 0
 targetTopic1       | true       | 3          | 1                  | 0         | 0
------------------------------------------------------------------------------------------------
ksql> show streams;

 Stream Name        | Kafka Topic  | Format
--------------------------------------------
 SOURCETOPIC2STREAM | sourceTopic2 | JSON
 TARGETTOPIC1STREAM | targetTopic1 | JSON
 SOURCETOPIC1STREAM | sourceTopic1 | JSON
--------------------------------------------
ksql>

Answer 1

这是KSQL中的错误。我已经在这里写下：https://github.com/confluentinc/ksql/issues/2123

解决方法是不要在kafka_topic中指定CREATE STREAM … AS：

ksql>
ksql> CREATE STREAM TargetStream WITH (partitions=3) AS select 'sourceTopic1' topicname, category, source, type, id, payload from sourceTopic1Stream where id like 'myid%';

 Message
----------------------------
 Stream created and running
----------------------------
ksql> INSERT INTO TargetStream SELECT 'sourceTopic2' AS topicname, * FROM sourceTopic2Stream where id like 'myid%';

 Message
-------------------------------
 Insert Into query is running.
-------------------------------
ksql> SELECT * FROM TargetStream;
1541496149897 | null | sourceTopic2 | Foo2 | bar: | x | myid2 | asdf
1541496141671 | null | sourceTopic1 | Foo1 | bar: | x | myid1 | asdf

ksql-下沉主题XXXX在元存储中不存在

1 个答案: