Question

对于flink来说是全新的。

试图实现简单的存储桶创建文件后，文件仍处于pending状态

case class Request (qname: String, rcode: String, ts:Long)
object Request extends DefaultJsonProtocol {
  implicit val dnsRequestFormat: RootJsonFormat[Request] = jsonFormat3(Request.apply)
}
val stream = env.fromElements(Request("aaa", "nothing", 1L), Request("bbb", "na", 2))
  val writer = new ParquetSinkWriter[Request]
  val bucketingSink = new BucketingSink[Request]("streaming/")
  bucketingSink.setBucketer(new DateTimeBucketer[Request]("yyyy-MM-dd--HHmm"))
  bucketingSink.setBatchRolloverInterval(1 * 30 * 1000); // this is 30 sec
  stream.addSink (bucketingSink)
  env.execute()

尝试使用SequenceFileWriter，但编译失败（此操作只是从doc复制而来，但编译失败（在setWriter行上）

val sink = new BucketingSink[String]("/base/path")
  sink.setBucketer(new DateTimeBucketer[String]("yyyy-MM-dd--HHmm"))
  sink.setWriter(new SequenceFileWriter[IntWritable, Text]())
  sink.setBatchSize(1024 * 1024 * 400) // this is 400 MB,
  sink.setBatchRolloverInterval(20 * 60 * 1000); // this is 20 mins

Error:(10, 18) type mismatch;
 found   : org.apache.flink.streaming.connectors.fs.SequenceFileWriter[org.apache.hadoop.io.IntWritable,org.apache.hadoop.io.Text]
 required: org.apache.flink.streaming.connectors.fs.Writer[String]
  sink.setWriter(new SequenceFileWriter[IntWritable, Text]())

bucketingSink文件保持挂起状态

0 个答案: