Question

我有问题。

我通过sbt原型创建了一个SCIO（Apache Beam）项目：sbt new spotify / scio.g8

这项工作的目标是从GS读取实木复合地板文件

当我直接在SCIO中使用Apache Beam提供的ParquetIO时，此工作（GenericRecord）：

val SCHEMA: Schema =
    new Schema.Parser()
      .parse(
        "{\n"
          + " \"namespace\": \"ioitavro\",\n"
          + " \"type\": \"record\",\n"
          + " \"name\": \"TestObject\",\n"
          + " \"fields\": [\n"
          + "     {\"name\": \"field1\", \"type\": \"int\"}\n,"
          + "     {\"name\": \"field2\", \"type\": \"int\"}\n"
          + " ]\n"
          + "}")

val inputIO = ParquetIO
 .read(SCHEMA)
 .from(options.getInputParquet)

sc.customInput("input", inputIO)
.map(file => file.toString)
.saveAsTextFile(args(outputTopic))

但是我想使用scio-parquet：https://spotify.github.io/scio/io/Parquet.html#read-avro-parquet-files

build.sbt文件中具有以下依赖项：

val scioVersion = "0.8.0-beta2"
val beamVersion = "2.16.0"
val scalaMacrosVersion = "2.1.1"

"com.spotify" %% "scio-core" % scioVersion,
"com.spotify" %% "scio-extra" % scioVersion,
"com.spotify" %% "scio-parquet" % scioVersion,
"com.spotify" %% "scio-avro" % scioVersion,

我使用模式

object TestModel {

  @AvroType.fromSchema(
    """{
      | "type":"record",
      | "name":"TestObject",
      | "namespace":"com.spotify.scio.avro",
      | "doc":"Record for an account",
      | "fields":[
      |   {"name":"field1","type":"int"},
      |   {"name":"field2","type":"int"}
      |  ]}
      """.stripMargin)
  class TestSchema

}

然后在我使用的工作中：

import com.spotify.scio.avro.TestObject

val projection = Projection[TestObject](_.getIntField, _.getIntField)

sc.parquetAvroFile[TestObject](args(inputParquet), projection)
  .map(file => file.toString)
  .saveAsTextFile(args(outputTopic))

但是我有以下错误：

object TestObject is not a member of package com.spotify.scio.avro
[error] import com.spotify.scio.avro.TestObject
[error]        ^
[error] /mypath/my-job/src/main/scala/com/test/MyJob.scala:41:33: not found: type TestObject
[error]     val projection = Projection[TestObject](_.getIntField, _.getIntField)
[error]                                 ^
[error] /my-path/my-job/src/main/scala/test/MyJob.scala:59:26: not found: type TestObject
[error]       sc.parquetAvroFile[TestObject](args(inputParquet), projection)
[error]                          ^

我不明白为什么，但是架构@ AvroType.fromSchema无法正确生成案例类TestObject。

或者我可能没有正确使用api，但是我点击了链接：https://spotify.github.io/scio/io/Parquet.html#read-avro-parquet-files

感谢您的帮助。

Answer 1

使用sbt-avro插件可以完成这项工作。

SCIO用Scio实木复合地板读取实木复合地板文件，未找到生成的案例类

1 个答案: