Question

我正在尝试对 Spark 进行一些转换，它在群集（YARN，Linux机器）上运行良好。但是，当我尝试在单元测试下在本地计算机（ Windows 7 ）上运行它时，我遇到了错误：

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)

我的代码如下：

@Test
def testETL() = {
    val conf = new SparkConf()
    val sc = new SparkContext("local", "test", conf)
    try {
        val etl = new IxtoolsDailyAgg() // empty constructor

        val data = sc.parallelize(List("in1", "in2", "in3"))

        etl.etl(data) // rdd transformation, no access to SparkContext or Hadoop
        Assert.assertTrue(true)
    } finally {
        if(sc != null)
            sc.stop()
    }
}

为什么要尝试访问hadoop？我该如何解决？提前谢谢

Answer 1

我已经在自己的http://simpletoad.blogspot.com/2014/07/runing-spark-unit-test-on-windows-7.html

上解决了这个问题

在Windows上运行spark单元测试

1 个答案: