Question

我的目标是我拥有具有 Avro 序列化属性的 spring boot应用程序kafka生产者，并且除了将消息推送到应由融合Sink Connector访问的相应主题之外，并插入到mysql / Oracle数据库表中，能够生成Avro序列化，并且 spring boot Consumer 可以对Avro进行反序列化，但是我的Sink连接器不起作用，我无法捕获到哪种有效负载接收器连接器除外，Spring Boot生产者应该如何编码以推送消息，使接收器连接器可以使用该属性

预先感谢：）

这是Spring Boot应用程序中的application.yml

server: 
    port: 9000
    spring.kafka:
    bootstrap-servers: "localhost:9092"
    properties:
      schema.registry.url: "http://localhost:8081"
      specific.avro.reader: true
    producer:
      key-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
      value-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
    app:
      topic: event_pay2

这是用于从Spring Boot应用程序创建模式的有效负载

{
  "schema": {
    "type": "struct",
    "fields": [
      {
        "type": "string",
        "optional": false,
        "field": "userid"
      },
      {
        "type": "string",
        "optional": false,
        "field": "regionid"
      },
      {
        "type": "string",
        "optional": false,
        "field": "gender"
      }
    ],
    "optional": false,
    "name": "oracle.user"
  },
  "payload": {
    "userid": "User_1",
    "regionid": "Region_5",
    "gender": "FEMALE"
  }
}

Pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.0.5.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.kafka</groupId>
<artifactId>kafka-producer-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-producer-example</name>
<description>Demo project for Spring Boot</description>

 <repositories>
    <repository>
        <id>confluent</id>
        <url>http://packages.confluent.io/maven</url>
    </repository>
</repositories>

<properties>
    <java.version>1.8</java.version>
    <confluent.version>4.0.0</confluent.version>
</properties>

<dependencies>
<dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
     <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-avro-serializer</artifactId>
        <version>${confluent.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.8.2</version>
</dependency>
</dependencies>

<build>
 <pluginManagement>
    <plugins>
    <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.8.2</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>schema</goal>
                </goals>
                <configuration>
                    <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
                    <outputDirectory>${project.build.directory}/generated/avro</outputDirectory>
                </configuration>
            </execution>
        </executions>
    </plugin>


        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
     </pluginManagement>
</build>

</project>

这是我剩下的电话，是如何将messgae推送到kafka主题中的

@PostMapping("/publish/avrodata")
public String sendMessage(@RequestBody String request) {
    sender.send(request);
    return "Published successfully";
}

最后我的水槽连接器

"name": "JDBC_Sink_EVENT_PAY",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "event_pay2",
"connection.url": "jdbc:mysql://localhost:3306/user",
"connection.user": "****",
"connection.password": "****",
"auto.create": "true",
"auto.evolve":"true",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://localhost:8081",
"key.converter.schemas.enable": "true",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081",
 "value.converter.schemas.enable": "true"

Answer 1

始终，始终在设置连接器之前调试主题。为此使用kafka-avro-console-consumer。如果那不起作用，则Connect + AvroConverter可能也不起作用（以我的经验），并且您可以减少问题空间。

如果我正确阅读了您的代码，则说明您发送的是String，而不是Avro对象

public String sendMessage(@RequestBody String request) {
    sender.send(request);  // <--- here and ^^^^^^ here
    return "Published successfully";
}

相反，您需要将输入请求解析为作为/src/main/resources/avro模式数据的一部分创建的对象，而不仅仅是将传入请求作为字符串转发。

该AVSC文件可能类似于

{
  "type": "Record",
  "namespace": "oracle.user",
  "name": "User",
  "fields": [
      { "type": "string", "name: "user_id" },
      { "type": "string", "name": "regionid" },
      { "type" "string", "name" "gender" }
   ]
}

将创建一个oracle.user.User对象，使您的KafkaTemplate必须类似于KafkaTemplate<String, oracle.user.User> sender

Kafka春季启动应用程序生产者，无法使用Avro格式的Kafka Sink Connecyor反映出来

1 个答案: