Question

我正在尝试使用zerocode

对流媒体平台（Kafka / spark）进行集成测试。

零代码（发布XML消息）->用Apache AVRO序列化-> Kafka主题->火花作业（处理）-> Kafka主题->用AVRO反序列化->零代码使用者测试（assert邮件内容）

当我试图以零代码将消息放在Kafka主题上时，零代码测试用例在XML消息开始处引入了特殊字符，这会在spark中解析XML消息时导致错误。

当我使用字符串序列化程序和“ hello world”时，它可以工作。

我需要使其与Apache Avro和XML消息一起使用。

{
  "scenario Name": "Produce a message to Kafka topic - vanilla",
  "steps": [
    {
      "name": "produce_step",
      "URL": "Kafka-topic: my-topic ",
      "operation": "produce",
      "request": {
        "record Type" : "J SON",
        "records": [
          {
            "key":"123456789",
            "value": "Schema Test"
          }
        ]
      },
      "assertions": {
        "record Metadata" : "$NOT.NULL",
        "status" : "OK"
      }
    }
  ]
}

当零代码测试用例在Kafka主题上放置消息时，不应在消息开始时引入任何特殊字符。

Answer 1

您只需要将“记录类型”更新为RAW，而不是JSON。它应该可以正常工作。

"record Type" : "RAW"

例如

您要向Kafka主题生成以下XML消息，然后使用Kafka主题中的XML并验证XML是否相同。

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>

下面是完整的测试用例，供您参考。它同时具有“生产”和“消费”步骤。

{
    "scenarioName": "Produce and Consume XML as RAW message",
    "steps": [
        {
            "name": "load_kafka",
            "url": "kafka-topic:demo-raw1",
            "operation": "load",
            "request": {
                "record Type" : "RAW",
                "records": [
                    {
                        "key": "${RANDOM.NUMBER}",
                        "value": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
                    }
                ]
            },
            "assertions": {
                "status": "Ok"
            }
        },
        {
            "name": "onload_kafka",
            "url": "kafka-topic:demo-raw1",
            "operation": "unload",
            "request": {
                "consumerLocalConfigs": {
                    "recordType": "RAW",
                    "commitSync": true,
                    "showRecordsConsumed": true,
                    "maxNoOfRetryPollsOrTimeouts": 3
                }
            },
            "assertions": {
                "size": 1,
                "records": [
                    {
                        "key" : "$NOT.NULL",
                        "value": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
                    }
                ]
            }
        }
    ]
}

在使用记录后，Zerocode测试运行控制台将在下面记录日志，即所使用的XML与生成的XML完全相同。

org.jsmart.zerocode.core.kafka.receive.KafkaReceiver - Consumer chosen recordType: RAW
2020-05-19 12:32:28,986 [main] INFO  
org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
Record Key - 6784799632078391362 , 
Record value - <?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>, 

Record partition - 0, Record offset - 1

您可以在下面看到完整的日志：


2020-05-19 12:32:25,358 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
-------------------------- BDD: Scenario:consume as RAW message -------------------------

2020-05-19 12:32:25,461 [main] INFO  org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
---------------------------------------------------------
kafka.bootstrap.servers - localhost:9092
---------------------------------------------------------
returned default type 'RAW'.
2020-05-19 12:32:25,677 [main] INFO  org.jsmart.zerocode.core.kafka.send.KafkaSender - Sending record number: 0

2020-05-19 12:32:25,678 [main] INFO  org.jsmart.zerocode.core.kafka.send.KafkaSender - Synchronous Producer sending record - ProducerRecord(topic=demo-raw1, partition=null, headers=RecordHeaders(headers = [], isReadOnly = false), key=6784799632078391362, value=<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ConversionRate xmlns="http://www.webserviceX.NET/">
      <FromCurrency>AFA</FromCurrency>
      <ToCurrency>GBP</ToCurrency>
    </ConversionRate>
  </soap:Body>
</soap:Envelope>, timestamp=null)


2020-05-19 12:32:25,873 [main] INFO  org.jsmart.zerocode.core.engine.validators.ZeroCodeValidatorImpl - Comparing results via LENIENT matchers
2020-05-19 12:32:25,878 [main] INFO  org.jsmart.zerocode.core.runner.StepNotificationHandler - 
***Step PASSED - Scenario:consume as RAW message -> load_kafka
2020-05-19 12:32:25,882 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
--------- TEST-STEP-CORRELATION-ID: a8ebe4c6-c1fb-434c-9c00-b2cbdf2f7fe6 ---------
*requestTimeStamp:2020-05-19T12:32:25.455
step:load_kafka
url:kafka-topic:demo-raw1
method:load
request:
{
  "record Type" : "RAW",
  "records" : [ {
    "key" : "6784799632078391362",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
  } ]
} 
--------- TEST-STEP-CORRELATION-ID: a8ebe4c6-c1fb-434c-9c00-b2cbdf2f7fe6 ---------
Response:
{
  "status" : "Ok",
  "recordMetadata" : {
    "offset" : 1,
    "timestamp" : 1589887945831,
    "serializedKeySize" : 19,
    "serializedValueSize" : 407,
    "topicPartition" : {
      "hash" : 749906548,
      "partition" : 0,
      "topic" : "demo-raw1"
    }
  }
}
*responseTimeStamp:2020-05-19T12:32:25.872 
*Response delay:417.0 milli-secs 
---------> Expected Response: <----------
{
  "status" : "Ok"
} 

-done-

2020-05-19 12:32:25,884 [main] INFO  org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper - 
---------------------------------------------------------
kafka.bootstrap.servers - localhost:9092
---------------------------------------------------------

***Step PASSED - Scenario:consume as RAW message -> onload_kafka
2020-05-19 12:32:33,025 [main] INFO  org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl - 
--------- TEST-STEP-CORRELATION-ID: 53092b7e-ae92-4631-ad79-0602280650b9 ---------
*requestTimeStamp:2020-05-19T12:32:25.884
step:onload_kafka
url:kafka-topic:demo-raw1
method:unload
request:
{
  "consumerLocalConfigs" : {
    "recordType" : "RAW",
    "commitSync" : true,
    "showRecordsConsumed" : true,
    "maxNoOfRetryPollsOrTimeouts" : 3
  }
} 
--------- TEST-STEP-CORRELATION-ID: 53092b7e-ae92-4631-ad79-0602280650b9 ---------
Response:
{
  "records" : [ {
    "topic" : "demo-raw1",
    "partition" : 0,
    "offset" : 1,
    "timestamp" : 1589887945831,
    "timestampType" : "CREATE_TIME",
    "serializedKeySize" : 19,
    "serializedValueSize" : 407,
    "key" : "6784799632078391362",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>",
    "leaderEpoch" : {
      "value" : 0
    }
  } ],
  "size" : 1
}
*responseTimeStamp:2020-05-19T12:32:33.020 
*Response delay:7136.0 milli-secs 
---------> Expected Response: <----------
{
  "size" : 1,
  "records" : [ {
    "key" : "$NOT.NULL",
    "value" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n  <soap:Body>\n    <ConversionRate xmlns=\"http://www.webserviceX.NET/\">\n      <FromCurrency>AFA</FromCurrency>\n      <ToCurrency>GBP</ToCurrency>\n    </ConversionRate>\n  </soap:Body>\n</soap:Envelope>"
  } ]
} 

-done-


2020-05-19 12:32:33,233 [main] INFO  org.jsmart.zerocode.core.domain.builders.ExtentReportsFactory - Where were the tests fired? Ans: OS:Mac OS X, Architecture:x86_64, Java:1.8.0_91, Vendor:Oracle Corporation

Process finished with exit code 0

以下是Kafka Produce Consume XML messages的Wiki和帮助文档

Answer 2

特殊字符在xml的开头

如果我理解正确，则您正在尝试将XML制成Avro ...

根据我对零码的了解，这是不可能的，并且至少要做的是将UTF-8字符串转换为Avro UTF-8字符串，您可以完全模拟同一件事在零代码之外。

启动架构注册表和Kafka

创建主题

kafka-topics --create --topic sample --zookeeper localhost:2181 --partitions=1 --replication-factor=1

使用kafka-avro-console-producer

为主题创建“ XML”字符串

kafka-avro-console-producer --broker-list localhost:29092 --topic sample --property value.schema='{"type":"string"}'
"<?xml version=\"1.0\"?>"

（可选）查看架构注册表以验证架构是否存在

curl -s localhost:8081/subjects/sample-value/versions/latest | jq
{
  "subject": "sample-value",
  "version": 1,
  "id": 29,
  "schema": "\"string\""
}

使用kafka-console-consumer（使用StringDeserializer）检查消息。 我们注意到它具有“特殊字符”
```
kafka-console-consumer --bootstrap-server localhost:9092 --topic sample --from-beginning
*<?xml version="1.0"?>
```
使用kafka-avro-console-consumer（使用KafkaAvroDeserializer），我们看到值用引号引起来（因为表示形式是JSON字符串），但是“特殊字符” 消失了。

kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic sample --from-beginning "<?xml version=\"1.0\"?>"

因此，我没有您的Spark代码来知道您的错误发生在哪里，但是spark-xml package对Avro数据一无所知（即它无法解析“特殊字符”，这实际上是一个注册表中的Avro ID）和spark-avro对模式注册表的使用方式一无所知，所以我的建议是忘掉Avro ，因为您实际上只在处理字符串。

将XML消息从Zerocode发布到Kafka主题

2 个答案: