Question

我有一个看起来像这样的数据框

<?php
    // Works as of PHP 5.3.0
    const CONSTANT = 'Hello World';
    echo CONSTANT;

    // Works as of PHP 5.6.0
    const ANOTHER_CONST = CONSTANT.'; Goodbye World';
    echo ANOTHER_CONST;

    const ANIMALS = array('dog', 'cat', 'bird');
    echo ANIMALS[1]; // outputs "cat"

    // Works as of PHP 7
    define('ANIMALS', array(
        'dog',
        'cat',
        'bird'
    ));
    echo ANIMALS[1]; // outputs "cat"
?>

我的架构看起来像这样 -

root
 |-- A1: string (nullable = true)
 |-- A2: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- A3 : string (nullable = true)
 |-- A4 : array (nullable = true)
 |    |-- element: string (containsNull = true)

我想将此数据帧转换为上面定义的模式。有人可以帮助我，我该怎么做？

注意： - 架构和数据框在运行时加载，而不是修复

Answer 1

您可以使用org.apache.spark.sql.expressions.UserDefinedFunction将字符串转换为数组，将arry转换为字符串，就像这样。

 val string_to_array_udf = udf((s:String) => Array(s))
 val array_to_string_udf = udf((a: Seq[String]) => a.head)
 val string_to_int_udf = udf((s:String) => s.toInt)

 val newDf = df.withColumn("a12", string_to_array_udf(col("a1"))).drop("a1").withColumnRenamed("a12", "a1")
 .withColumn("a32", string_to_int_udf(col("a3"))).drop("a3").withColumnRenamed("a32", "a3")
 .withColumn("a22", array_to_string_udf(col("a2"))).drop("a2").withColumnRenamed("a22", "a2")

 newDf.printSchema
 root
   |-- a4: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- a1: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- a3: integer (nullable = true)
   |-- a2: string (nullable = true)

使用spark中的不同模式从现有数据框创建另一个数据框

1 个答案: