Question

我正在努力将JavaRDD（字符串是JSON字符串）转换为数据帧并显示它。我正在做类似下面的事情，

public void call(JavaRDD<String> rdd, Time time) throws Exception {
            if (rdd.count() > 0) {
                JavaRDD<String> filteredRDD = rdd.filter(x -> x.length()>0);                    
                sqlContext = SQLContextSingleton.getInstance(filteredRDD.context());
                DataFrame df = sqlContext.read().schema(SchemaBuilder.buildSchema()).json(filteredRDD);
                df.show();
            }
        }

模式如下所示，

public static StructType buildSchema() {
    StructType schema = new StructType(
            new StructField[] { DataTypes.createStructField("student_id", DataTypes.StringType, false),
                    DataTypes.createStructField("school_id", DataTypes.IntegerType, false),
                    DataTypes.createStructField("teacher", DataTypes.StringType, true),
                    DataTypes.createStructField("rank", DataTypes.StringType, true),
                    DataTypes.createStructField("created", DataTypes.TimestampType, true),
                    DataTypes.createStructField("created_user", DataTypes.StringType, true),
                    DataTypes.createStructField("notes", DataTypes.StringType, true),
                    DataTypes.createStructField("additional_data", DataTypes.StringType, true),
                    DataTypes.createStructField("datetime", DataTypes.TimestampType, true) });
    return (schema);
}

上面的代码正在回复我，

|student_id|school_id|teacher|rank|created|created_user|notes|additional_data|datetime|
+----------+------+--------+-----+-----------+-------+------------+--------+-------------+-----+-------------------+---------+---------------+--------+----+-------+-----------+
|      null|  null|    null| null|       null|   null|        null|    null|         null|

但是，当我没有指定架构并将Dataframe创建为

时

DataFrame df = sqlContext.read().json(filteredRDD);

这给我的结果如下，

|student_id|school_id|teacher|rank|created|created_user|notes|additional_data|datetime|
        +----------+------+--------+-----+-----------+-------+------------+--------+-------------+-----+-------------------+---------+---------------+--------+----+-------+-----------+
        |      1|  123|    xxx| 3|       2017-06-02 23:49:10.410|   yyyy|        NULL|    good academics|         2017-06-02 23:49:10.410|

示例JSON记录：

{"student_id": "1","school_id": "123","teacher": "xxx","rank": "3","created": "2017-06-02 23:49:10.410","created_user":"yyyy","notes": "NULL","additional_date":"good academics","datetime": "2017-06-02 23:49:10.410"}

对我做错的任何帮助？

Answer 1

问题在于，在我的json记录中，school_id是字符串类型，而spark明确地无法从String转换为Integer。在这种情况下，它将整个记录视为null。我修改了我的模式，将school_id表示为StringType，解决了我的问题。在http://blog.antlypls.com/blog/2016/01/30/processing-json-data-with-sparksql/

提供了一些很好的解释

Spark Dataframe在指定Schema时返回NULL

1 个答案: