Question

我正在阅读固定位置文件。文件的最终结果存储在字符串中。我想将字符串转换为DataFrame以进一步处理。请帮助我。以下是我的代码：

输入数据： + --------- + ---------------------- +

| PRGREFNBR |值|

+ --------- + ---------------------- +

| 01 | 11 apple TRUE 0.56 |

| 02 | 12 pear FALSE1.34 |

| 03 | 13 raspberry TRUE 2.43 |

| 04 | 14 plum TRUE .31 |

| 05 | 15 cherry TRUE 1.4 |

+ --------- + ---------------------- +

数据位置："3,10,5,4"

数据框中默认标头的预期结果：

+ ----- + + ----- + ---------- + ----- ----- +

|的SeqNo | col_0 | COL_1 | COL_2 | col_3 |

+ ----- + + ----- + ---------- + ----- ----- +

| 01 | 11 | apple | TRUE | 0.56 |

| 02 | 12 |梨| FALSE | 1.34 |

| 03 | 13 |覆盆子|真实| 2.43 |

| 04 | 14 |李子|真实| 1.31 |

| 05 | 15 |樱桃|真实| 1.4 |

+ ----- + + ----- + ---------- + ----- ----- +

Answer 1

给定固定位置文件（比如input.txt）：

11 apple     TRUE 0.56

12 pear      FALSE1.34 

13 raspberry TRUE 2.43 

14 plum      TRUE 1.31 

15 cherry    TRUE 1.4

和输入文件中每个字段的长度为（比如lengths）：

3,10,5,4

您可以按如下方式创建DataFrame：

// Read the text file as is
// and filter out empty lines
val lines = spark.read.textFile("input.txt").filter(!_.isEmpty)

// define a helper function to do the split per fixed lengths
// Home exercise: should be part of a case class that describes the schema
def parseLinePerFixedLengths(line: String, lengths: Seq[Int]): Seq[String] = {
  lengths.indices.foldLeft((line, Array.empty[String])) { case ((rem, fields), idx) =>
    val len = lengths(idx)
    val fld = rem.take(len)
    (rem.drop(len), fields :+ fld)
  }._2
}

// Split the lines using parseLinePerFixedLengths method
val lengths = Seq(3,10,5,4)
val fields = lines.
  map(parseLinePerFixedLengths(_, lengths)).
  withColumnRenamed("value", "fields") // <-- it'd be unnecessary if a case class were used
scala> fields.show(truncate = false)
+------------------------------+
|fields                        |
+------------------------------+
|[11 , apple     , TRUE , 0.56]|
|[12 , pear      , FALSE, 1.34]|
|[13 , raspberry , TRUE , 2.43]|
|[14 , plum      , TRUE , 1.31]|
|[15 , cherry    , TRUE , 1.4 ]|
+------------------------------+

您可能已经拥有的内容让我们将嵌套的字段序列展开/解构为列

val answer = lengths.indices.foldLeft(fields) { case (result, idx) =>
  result.withColumn(s"col_$idx", $"fields".getItem(idx))
}
// drop the unnecessary/interim column
scala> answer.drop("fields").show
+-----+----------+-----+-----+
|col_0|     col_1|col_2|col_3|
+-----+----------+-----+-----+
|  11 |apple     |TRUE | 0.56|
|  12 |pear      |FALSE| 1.34|
|  13 |raspberry |TRUE | 2.43|
|  14 |plum      |TRUE | 1.31|
|  15 |cherry    |TRUE | 1.4 |
+-----+----------+-----+-----+

完成！

如何从给定字段长度的固定长度文本文件创建DataFrame？

1 个答案: