Question

我正在尝试在Spring Cloud Data Flow中配置DLQ。这是Stream defination以及我如何部署它

  stream create --definition ":someTestTopic > custom-transform
     --spring.cloud.stream.bindings.input.consumer.headerMode=raw | log --spring.cloud.stream.bindings.input.consumer.headerMode=raw" --name ticktran


    stream deploy ticktran --properties
  "apps.log.spring.cloud.stream.bindings.input.consumer.headerMode=raw,apps.custom-transform.spring.cloud.stream.bindings.input.consumer.headerMode=raw,app.custom-transform.spring.cloud.stream.bindings.output.destination=test-tran,app.log.spring.cloud.stream.bindings.input.destination=test-tran,app.custom-transform.spring.cloud.stream.kafka.bindings.test-tran.consumer.enableDlq=true"

在自定义转换 - 处理器代码中，我提到了

if(out.contains("ERROR")) {
            throw new RuntimeException("Error ");
        }

这意味着如果消息包含ERROR然后RunTimeException，我想在DLQ中捕获这些消息。但是，当我运行代码时，我似乎没有得到任何名为test-tran的Kafka DL队列。

我是否需要设置更多属性来启用DLQ，或者我需要在代码中更改某些内容以正确使用DLQ。

自定义转换代码

TransformationServiceApplication.java

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.hateoas.config.EnableEntityLinks;

@SpringBootApplication
@EnableEntityLinks
public class TransformationServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(TransformationServiceApplication.class, args);
    }
}

TransformationMessageEndPoint.java

@EnableBinding(Processor.class)
@MessageEndpoint
public class TransformationMessageEndpoint {

    private static final String NS = "http://openrisk.com/ingestion/";

    AtomicInteger index = new AtomicInteger(1);
    @ServiceActivator(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
    public Object process(Message<?> message) {
        String out =  new String((byte[])message.getPayload());

        System.out.println("*****" + out);

        if(out.contains("ERROR")) {
            throw new RuntimeException("Error ");
        }

        return message;

    }
}

的pom.xml

<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.6.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-dataflow-server-local</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-stream-kafka</artifactId>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-stream-binder-kafka -->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream-binder-kafka</artifactId>
            <version>1.0.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream-test-support</artifactId>
            <version>1.0.0.BUILD-SNAPSHOT</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud.stream.module</groupId>
            <artifactId>spring-cloud-stream-modules-test-support</artifactId>
            <version>1.0.0.BUILD-SNAPSHOT</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.jena</groupId>
            <artifactId>jena-core</artifactId>
            <version>3.1.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.8.0</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

添加模块

app register --name custom-transform --type processor --uri maven://com.openrisk.openmargin:TransformationService:0.0.1-SNAPSHOT

添加流

stream create --definition ":someTesstTopic > custom-transform | log " --name ticktran

部署流

stream deploy ticktran --properties "app.log.spring.cloud.stream.bindings.input.consumer.headerMode=raw,app.custom-transform.spring.cloud.stream.bindings.input.consumer.headerMode=raw,app.custom-transform.spring.cloud.stream.kafka.bindings.input.consumer.enableDlq"

Answer 1

您的流定义几乎没有问题。

部署属性以app.<app-name>.开头，但您在很少的地方apps.<app-name>.。
目的地是在SCDF中自动创建的，因此不建议覆盖默认值。但是，当您运行独立的spring-cloud-stream应用程序时，您可以执行此操作。
您可以通过直接与默认渠道进行交互来启用DLQ，而不是使用自定义目的地 - 请参阅下面的示例。

stream create --definition＆＃34;：someTesssstTopic＆gt;变换| log＆＃34; --name ticktran

stream deploy ticktran --properties＆＃34; app.log.spring.cloud.stream.bindings.input.consumer.headerMode = raw，app.transform.spring.cloud.stream.bindings.input.consumer.headerMode =原料，app.transform.spring.cloud.stream.kafka.bindings.input.consumer.enableDlq＆＃34;

目标test-tran在引用app.transform.spring.cloud.stream.kafka.bindings.<channelName>.consumer.enableDlq属性时不符合可接受的格式。
最后，只有在出现错误时才会创建error.<destination>.<group>主题。

我们将通过以下方式向参考指南添加一些DSL样本：#885。

修改我更新了流定义以反映正确的部署属性前缀。

Answer 2

我使用下面提到的命令将数据流版本更改为1.1 M1 release以创建和部署属性，它现在正在运行

stream create --definition ":someTesstTopic > transform | log " --name ticktran


stream deploy ticktran --properties "app.log.spring.cloud.stream.bindings.input.consumer.headerMode=raw,app.transform.spring.cloud.stream.bindings.input.consumer.headerMode=raw,app.transform.spring.cloud.stream.kafka.bindings.input.consumer.enableDlq"

感谢Sabby Anandan

Spring Cloud数据流DLQ配置无法正常工作

2 个答案: