Question

正在寻找有关此错误的帮助，对于通过此错误的任何建议，我将非常感谢。

正如主题所说，我正在尝试运行一个非常简单的pyspark任务，但是却收到“ OSError：[Errno 2] No such file or directory”

我的工作目录中有...

luigi.cfg
sparkLuigiTest.py

我的配置文件包含...

[spark]
spark-submit:$SPARK_HOME/bin/spark-submit
master:yarn
num-executors:10

*。py文件包含...

import luigi
from luigi.contrib.spark import PySparkTask

class Test(PySparkTask):
    def input(self):
        return None

    def output(self):
        return luigi.LocalTarget("output.csv")

    def main(self, sc, *args):
        data = [(1,2,3), (4,5,6), (7,8,9)]
        data = sc.createDataFrame(data, ["A", "B", "C"])

        df = data.toPandas()
        df.to_csv("/u/<user name>/tests/luigi_woo.csv")

if __name__ == "__main__":
    luigi.run()

最后，我使用

运行脚本

LUIGI_CONFIG_PATH=./luigi.cfg python ./sparkLuigiTest.py --local-scheduler Test

关于为什么会出现以下错误的任何想法？

ERROR: [pid 1678418] Worker Worker(salt=778216410, workers=1, host=hpchdp2e, username=<name>, pid=1678418) failed    Test()
Traceback (most recent call last):
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/site-packages/luigi/contrib/spark.py", line 309, in run
    super(PySparkTask, self).run()
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/site-packages/luigi/contrib/spark.py", line 66, in run
    super(SparkSubmitTask, self).run()
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/site-packages/luigi/contrib/external_program.py", line 134, in run
    with self._proc_with_tracking_url_context(proc_args=args, proc_kwargs=kwargs) as proc:
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/site-packages/luigi/contrib/external_program.py", line 168, in _proc_with_tracking_url_context
    main_proc = subprocess.Popen(proc_args, **proc_kwargs)
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/s/anaconda/users/<name>/miniconda2/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Luigi：提交PySpark任务时没有此类文件或目录错误

0 个答案: