Question

我有很多文本文件，其中填充了不同类型的日志消息，但每个文件只能显示一种类型的消息。

File1：（I）; 2017-01-12; 16：54：45;随机文本;其他文本
文件2：（I）; 2017-01-13 15:34:56;再次发短信;再多一次//i.e。空间日期和时间之间

我已经开始工作，但我想问一下这是不是正确的＃34;这样做的方式。此外，只有当分号和空格之间的变化总是出现在同一位置时，我的方法才有效关于此事的任何建议都表示赞赏，因为我是scala / spark的新手。

//read file
val df = spark.read.textFile(file.path).filter(f => f.nonEmpty && f.length > 1 && f.startsWith("("))
//create empty dataset of type OutputMessage
var df3 = Seq.empty[OutputMessage].toDS()
//get number of semicolons within first line of the dataset to determine type
val message_type = df.take(1).mkString(",").count(_ == ';')

if(message_type == 5){
    //split by semicolon and create dataset of type InputMessage
    var df2 = df.map(x => x.split(";")).map(x => InputMessage(x(0), x(1), x(2), x(3), x(4), x(5)))
    //map to dataset of type output message
    df3 = df2.map(
      x =>
        OutputMessage(x.status,
          x.messages_datestring,
          x.messages_timestring,
          x.device,
          x.device_fullmessage,
          x.device_message,
          fileName,
          getWeekday(x.messages_datestring),
          (x.messages_datestring + "T" + x.messages_timestring),
          data_company,
          data_location,
          data_systemname)
    )
  }
  else if (message_type == 4){
    var df2 = df.map(x => x.split(";")).map(x => InputMessage1(x(0), x(1), x(2), x(3), x(4)))
    df3 = df2.map(
      x=>
        OutputMessage(x.status,
          x.messages_datetimestring.split(" ").take(1).mkString(","),
          x.messages_datetimestring.split(" ").takeRight(1).mkString(","),
          x.device,
          x.device_fullmessage,
          x.device_message,
          fileName,
          getWeekday(x.messages_datetimestring.split(" ").take(1).mkString(",")),
          x.messages_datetimestring.replace(' ', 'T'),
          data_company,
          data_location,
          data_systemname)
    )
  }
//convert to rdd
val dsToRDD = df3_filtered.rdd
//laod to elasticsearch
dsToRDD.saveToEs("abdata/log")

编辑：我刚看到一些文件在行之间有不一致。这意味着我的解决方案不再适用了

编辑：将其更改为基于行的执行。到目前为止，大多数事情都有效，除了行内的随机分隔符。我得到了这个案例的输出但不是想要的。

  object MapRawData{
  def mapRawLine (line: String): Option[RawMessage] ={
    var msgtype = 0;
    val fields = line.split(";")
    if (fields(0).length == 3 && fields(1).length == 10) msgtype = 1
    if (fields(0).length == 3 && fields(1).length > 10) msgtype = 3
    if (fields(0).length > 16) msgtype = 2
    try {
      fields.map(_.trim)
      Some(
        RawMessage(
          status = fields(0).take(3),
          messages_datestring = if(msgtype == 1) fields(1) else if(msgtype == 2) fields(0).drop(4).take(10) else fields(1).take(10),
          messages_timestring = if(msgtype == 1) fields(2).take(8) else if (msgtype == 2) fields(0).drop(15).take(8) else (fields(1).drop(11).take(8)),
          device = if(msgtype == 1) fields(3) else if (msgtype == 2) fields(1) else fields(2),
          device_fullmessage = if(msgtype == 1) fields(4) else if (msgtype == 2) fields(2) else fields(3),
          device_message = if(msgtype == 1) fields(5) else if (msgtype == 2) fields(3) else fields(4)
        )
      )
    }
    catch {
      case e: Exception =>
        println(s"Unable to parse line: $line")
        None
    }
  }
}

这种变化的方式/资源消耗比第一种方式更多吗？

Answer 1

将其更改为基于行的执行。到目前为止，大多数事情都有效，除了行内的随机分隔符。我得到了这个案例的输出但不是想要的。

object MapRawData{
  def mapRawLine (line: String): Option[RawMessage] ={
    var msgtype = 0;
    val fields = line.split(";")
    if (fields(0).length == 3 && fields(1).length == 10) msgtype = 1
    if (fields(0).length == 3 && fields(1).length > 10) msgtype = 3
    if (fields(0).length > 16) msgtype = 2
    try {
      fields.map(_.trim)
      Some(
        RawMessage(
          status = fields(0).take(3),
          messages_datestring = if(msgtype == 1) fields(1) else if(msgtype == 2) fields(0).drop(4).take(10) else fields(1).take(10),
          messages_timestring = if(msgtype == 1) fields(2).take(8) else if (msgtype == 2) fields(0).drop(15).take(8) else (fields(1).drop(11).take(8)),
          device = if(msgtype == 1) fields(3) else if (msgtype == 2) fields(1) else fields(2),
          device_fullmessage = if(msgtype == 1) fields(4) else if (msgtype == 2) fields(2) else fields(3),
          device_message = if(msgtype == 1) fields(5) else if (msgtype == 2) fields(3) else fields(4)
        )
      )
    }
    catch {
      case e: Exception =>
        println(s"Unable to parse line: $line")
        None
    }
  }
}

将具有不同格式的文本文件映射到数据集

1 个答案: