Question

我尝试使用java和apache spark 1.0.0版本为决策树分类器实现简单演示。我以http://spark.apache.org/docs/1.0.0/mllib-decision-tree.html为基础。到目前为止，我已经编写了下面列出的代码。

符合以下代码我收到错误：

org.apache.spark.mllib.tree.impurity.Impurity impurity = new org.apache.spark.mllib.tree.impurity.Entropy();

类型不匹配：无法从Entropy转换为Impurity。这对我来说很奇怪，而类Entropy实现了Impurity接口：

https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/mllib/tree/impurity/Entropy.html

我正在寻找答案，为什么我无法完成这项任务？

package decisionTree;

import java.util.regex.Pattern;

import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.mllib.regression.LabeledPoint;
import org.apache.spark.mllib.tree.DecisionTree;
import org.apache.spark.mllib.tree.configuration.Algo;
import org.apache.spark.mllib.tree.configuration.Strategy;
import org.apache.spark.mllib.tree.impurity.Gini;
import org.apache.spark.mllib.tree.impurity.Impurity;

import scala.Enumeration.Value;

public final class DecisionTreeDemo {

    static class ParsePoint implements Function<String, LabeledPoint> {
        private static final Pattern COMMA = Pattern.compile(",");
        private static final Pattern SPACE = Pattern.compile(" ");

        @Override
        public LabeledPoint call(String line) {
            String[] parts = COMMA.split(line);
            double y = Double.parseDouble(parts[0]);
            String[] tok = SPACE.split(parts[1]);
            double[] x = new double[tok.length];
            for (int i = 0; i < tok.length; ++i) {
                x[i] = Double.parseDouble(tok[i]);
            }
            return new LabeledPoint(y, Vectors.dense(x));
        }
    }

    public static void main(String[] args) throws Exception {

        if (args.length < 1) {
            System.err.println("Usage:DecisionTreeDemo <file>");
            System.exit(1);
        }

        JavaSparkContext ctx = new JavaSparkContext("local[4]", "Log Analizer",
                System.getenv("SPARK_HOME"),
                JavaSparkContext.jarOfClass(DecisionTreeDemo.class));

        JavaRDD<String> lines = ctx.textFile(args[0]);
        JavaRDD<LabeledPoint> points = lines.map(new ParsePoint()).cache();

        int iterations = 100;

        int maxBins = 2;
        int maxMemory = 512;
        int maxDepth = 1;

        org.apache.spark.mllib.tree.impurity.Impurity impurity = new org.apache.spark.mllib.tree.impurity.Entropy();

        Strategy strategy = new Strategy(Algo.Classification(), impurity, maxDepth,
                maxBins, null, null, maxMemory);

        ctx.stop();
    }
}

@samthebest如果我删除杂质变量并改为以下形式：

Strategy strategy = new Strategy(Algo.Classification(), new org.apache.spark.mllib.tree.impurity.Entropy(), maxDepth, maxBins, null, null, maxMemory);

错误更改为：构造函数Entropy（）未定义。

[编辑] 我发现我认为正确调用方法（https://issues.apache.org/jira/browse/SPARK-2197）：

Strategy strategy = new Strategy(Algo.Classification(), new Impurity() {
@Override
public double calculate(double arg0, double arg1, double arg2)
{ return Gini.calculate(arg0, arg1, arg2); }

@Override
public double calculate(double arg0, double arg1)
{ return Gini.calculate(arg0, arg1); }

}, 5, 100, QuantileStrategy.Sort(), null, 256);

不幸的是我遇到了错误:(

Answer 1

Bug 2197的Java解决方案现已通过this pull request：

提供

对Java易于使用的决策树的其他改进： *杂质类：添加了实例（）方法来帮助Java接口。 *策略：添加了Java友好的构造函数 - ＆GT;注意：我从Java友好的构造函数中删除了quantileCalculationStrategy，因为（a）它是一个特殊类，（b）只有1 选项目前。我怀疑我们会在另一个之前重做API 选项包括在内。

你可以看到一个完整的例子，即使用Gini杂质的intance（）方法来解决你的问题here

Strategy strategy = new Strategy(Algo.Classification(), Gini.instance(), maxDepth, numClasses,maxBins, categoricalFeaturesInfo);
DecisionTreeModel model = DecisionTree$.MODULE$.train(rdd.rdd(), strategy);

使用java在apache spark中的决策树实现问题

1 个答案: