Question

我有数据集df，其内容的索引为accountid，我也有一个带有accountids的数组列表。如何过滤或映射数据集以创建新数据集，该数据集仅包含基于arraylist中accountid的内容。

我正在使用Java 8

List<String> accountIdList= new ArrayList<String>();
accountIdList.add("1001");
accountIdList.add("1002");
accountIdList.add("1003");
accountIdList.add("1004");
Dataset<Row> filteredRows=  df.filter(p-> df.col("accountId").equals(accountIdList));

我试图为比较运算符传递列表本身你认为这是正确的方法

Java语法是

如果您正在寻找java语法

Dataset<Row> filteredRows=  df.where(df.col("accountId").isin(accountIdList.toArray()));

Answer 1

使用Column.isin方法：

import scala.collection.JavaConversions;
import static org.apache.spark.sql.functions.*;

Dataset<Row> filteredRows = df.where(col("accountId").isin(
  JavaConversions.asScalaIterator(accountIdList.iterator()).toSeq()
));

Answer 2

这是Java中的工作代码。希望它有所帮助。

这是我的sampleFile内容（输入）： -

1001

1008

1005

1009

1010

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;

import org.apache.spark.api.java.function.FilterFunction;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.SparkSession;

public class DatasetFilter {

    private static List<String> sampleList = new ArrayList<String>();

    public static void main(String[] args)
    {
        sampleList.add("1001");
        sampleList.add("1002");
        sampleList.add("1003");
        sampleList.add("1004");
        sampleList.add("1005");

        SparkSession sparkSession = SparkSession.builder()
                .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
                .config("spark.sql.warehouse.dir", "file:///C:/Users/user/workspace/Validation/spark-warehouse")
                .master("local[*]").getOrCreate();

        //Read the source-file.
        Dataset<String> src = sparkSession.read().textFile("C:\\Users\\user\\Desktop\\dataSetFilterTest.txt");
        src.show(10);

        //Apply filter
        Dataset<String> filteredSource = src.filter(new FilterFunction<String>() {

            private static final long serialVersionUID = 1L;

            @Override
            public boolean call(String value) throws Exception
            {
                System.out.println("***************************************");
                boolean status = false;
                Iterator<String> iterator = sampleList.iterator();
                while (iterator.hasNext()) {
                    String val = iterator.next();
                    System.out.println("Val is :: " + val + " Value is :: " + value);
                    if (value.equalsIgnoreCase(val)) {
                        status = true;
                        break;
                    } else {
                        status = false;
                    }
                }
                return status;
            }
        });

        filteredSource.show();

        System.out.println("Completed the job :)");
    }

}

输出： -

基于索引id的ArrayList过滤JavaRDD

2 个答案: