通过Kafka连接器从文件流式传输

时间:2019-08-08 15:52:22

标签: apache-kafka apache-kafka-connect

  • 用例:

    尝试从文件流式传输数据。

  • 引荐链接:

    https://docs.confluent.io/current/connect/kafka-connect-spooldir/connectors/examples/csv_source_connector_with_schema_example.html

  • 假脱机目录配置:

    name = CsvSchemaSpoolDir     task.max = 1     connector.class = com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourc eConnector     input.path = / usr / hdp / 3.0.1.0-187 / confluentclient /     input.file.pattern = csv-spooldir-source.csv     error.path = / usr / hdp / 3.0.1.0-187 / confluentclient /错误     finish.path = / usr / hdp / 3.0.1.0-187 / confluentclient /完成     halt.on.error =假     topic = spooldir-testing-topic     csv.first.row.as.header = true     key.schema = {\ n \“ name \”:\“ com.example.users.UserKey \”,\ n \“ type \”:\“ STRUCT \”,\ n \“ isOptional \”:false,\ n \“ fieldSchemas \”:{\ n \“ id \”:{\ n \“ type \”:\“ INT64 \”,\ n \“ isOptional \”:false \ n} \ n} \ n}     value.schema = {\ n \“ name \”:\“ com.example.users.User \”,\ n \“ type \”:\“ STRUCT \”,\ n \“ isOptional \”:false,\ n \“ fieldSchemas \”:{\ n \“ id \”:{\ n \“ t ype \”:\“ INT64 \”,\ n \“ isOptional \”:false \ n},\ n \“ first_name \“:{\ n \” type \“:\” STRING \“,\ n \” isOptional \“:true \ n},\ n \”姓氏“:{\ n \” type \“:\ “ STRING \”,\ n \“ isOptional \”:true \ n},\ n \“电子邮件\”:{\ n \“ type \”:\“ STRING \”,\ n \“ isOptional \”:true \ n},\ n \“性别\”:{\ n \“类型\”:\“ STRING \”,\ n \“ isOptional \”:消息\ n},\ n \“ ip_address \”:{ \ n \“类型\”:\“ STRING \ “,\ n \” isOptialal \“:true \ n},\ n \” last_login \“:{\ n \” type \“:\” STRING \“,\ n \” isOptional \“:true \ n },\ n \“ account_balance \”:{\ n \“ name \”:\“ org.apache.kafka.connect.data.Decimal \”,\ n \“ type \”:\“ BYTES \”,\ n \“版本\”:1,\ n \“参数\”:{\ n \“ scale \”:\“ 2 \” \ n},\ n \“ isOptional \”:true \ n},\ n \“ country \”:{\ n \“ type \”:\“ STR ING \”,\ n \“ isOptional \”:true \ n},\ n \“ favorite_color \”:{\ n \“ type \ “:\” STRING \“,\ n \” isOptional \“:true \ n} \ n} \ n}

  • 已执行的命令:

    bin / confluent load spooldir-spooldir-testing-topic -d /usr/hdp/3.0.1.0-187/confluentclient/confluent-5.2.2/share/confluent-hub-components/jcustenborder-kafka-connect- spooldir / etc / spooldir.properties

  • 问题:

    垃圾箱/汇合状态

    控制中心为[DOWN] ksql-server是[DOWN] 连接为[DOWN] kafka-rest是[UP] 模式注册表为[UP] kafka是[UP] 动物园管理员是[UP]

connect is [DOWN]服务已关闭且未启动。

  • 问题02:

    root@masbidw1.usa.corp.ad:/usr/hdp/3.0.1.0-187/confluentclient/confluent-5.2.2> bin / confluent加载spooldir-spooldir-testing-topic -d / usr / hdp /3.0.1.0-187/confluentclient/confluent-5.2.2/share/confluent-hub-components/jcustenborder-kafka-connect-spooldir/etc/spooldir.properties 此CLI仅用于开发,而不用于生产 https://docs.confluent.io/current/cli/index.html

    为'load'赋予了无效的参数'-'。

如何形成此命令:

confluent local load spooldir -- -d spooldir.properties

0 个答案:

没有答案