使用Scalavro和Parquet-avro编写Parquet文件

时间:2014-03-11 17:25:59

标签: scala avro parquet

我需要在镶木地板文件格式中写一个文件,以便用spark读取。 我使用scala与Scalavro和Parquet-avro。

在我的测试中,我编写了一个avro格式的文件并且工作正常:

import java.io._
import com.gensler.scalavro.types.AvroType
import scala.util.{ Success, Failure }

//object structure
case class defMyList(mydata:String)
case class objectTest(name: String, desc:String,myList:Seq[defMyList])

def test()
{

    //create object data 
    val objectList = objectTest(
      name = "object name",
      desc = "object desc",
      myList = Seq(
        defMyList("asdfasdfasfsafsdfasdfasdf"),
        defMyList("asdfasdfasfsafsdfasdfasdf")
      )
    )


    val objectListType = AvroType[objectTest]

    println("schema: " + objectListType.schema)

    val filestream= new File("C:\\avrofile.avro")
    val outStream = new FileOutputStream(filestream)
    objectListType.io.write(objectList, outStream)

    val inStream: java.io.InputStream = new FileInputStream(filestream)

    objectListType.io.read(inStream) match {
      case Success(readResult) => println("Successfully deserialized: " + readResult)
      case Failure(cause)      => println("Failure")
    }

}

如何更改此代码以写入拼花格式?

谢谢

0 个答案:

没有答案