Question

我是新来的火花并尝试学习它。我正在尝试使用一个类从textFile创建一个数据集。当我执行dataset.show（）时，它显示所有空白，列长度显示为0。

代码：

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class DatasetExample {

    public static void main(String[] args) {
        test(fileName);
    }

    static final String fileName = "inputFile";
    static SparkConf conf = new SparkConf().setMaster("local").setAppName("Test");
    static JavaSparkContext sc = new JavaSparkContext(conf);
    static SparkSession session = SparkSession.builder().config(conf).getOrCreate();

    private static void test(String fileName){
        JavaRDD<Input> rdd = sc.textFile(fileName).map(new Function<String, Input>() {
            @Override
            public Input call(String s) throws Exception {
                String[] str = s.split(",");
                System.out.println(str[0] + " and " + str[1] + " and " + str[2]);
                return new Input(str[0], str[1], Integer.parseInt(str[2]));
            }
        });
        Dataset<Row> dataSet = session.createDataFrame(rdd, Input.class);
        dataSet.show();
        System.out.println("Column length is: " + dataSet.columns().length);

    }

    static class Input{
        String key;
        String value;
        int number;

        Input(String key, String value, int number){
            this.key = key;
            this.value = value;
            this.number = number;
        }
    }
}

显示的输出是：

foo and A and 1
foo and A and 2
foo and A and 1
foo and B and 2
foo and B and 1
bar and C and 2
bar and D and 3
dek and X and 3
max and X and 3
eer and P and 3

++
||
++
||
||
||
||
||
||
||
||
||
||
++

Column length is: 0

我不想显式定义架构，但希望它从类结构中获取架构。我可能会缺少什么？

Answer 1

来自JavaBeans Wiki Definition：

在基于Java平台的计算中，JavaBean是可以将许多对象封装到一个对象（bean）中。他们是可序列化，具有零参数构造函数，并允许访问使用getter和setter方法的属性

因此，将其公开并生成getter / setter：

public static class Input {
    String key;
    String value;
    int number;

    public String getKey() {
        return key;
    }

    public void setKey(String key) {
        this.key = key;
    }

    public String getValue() {
        return value;
    }

    public void setValue(String value) {
        this.value = value;
    }

    public int getNumber() {
        return number;
    }

    public void setNumber(int number) {
        this.number = number;
    }

    public Input(String key, String value, int number) {
        this.key = key;
        this.value = value;
        this.number = number;
    }
}

您将获得输出。

数据集未显示任何列

1 个答案: