我一直在为此苦苦挣扎。
使用python3运行我的spark作业正常
> export PYSPARK_DRIVER_PYTHON=python3
> spark-submit myscript.py --help
usage: myscript.py [-h] -s START_DATE [-e END_DATE]
[--end_date_buffer END_DATE_BUFFER]
[--input INPUT] [--output OUTPUT]
切换到Ipython会破坏一切
> export PYSPARK_DRIVER_PYTHON=ipython
> spark-submit myscript.py --help
2018-08-20 12:16:31 WARN Utils:66 - Your hostname, owl resolves to a loopback address: 127.0.1.1; using 10.254.1.130 instead (on interface wlp2s0)
2018-08-20 12:16:31 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-08-20 12:16:31 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
=========
IPython
=========
Tools for Interactive Computing in Python
=========================================
A Python shell with automatic history (input and output), dynamic object
introspection, easier configuration, command completion, access to the
system shell and more. IPython can also be embedded in running programs.
Usage
ipython [subcommand] [options] [-c cmd | -m mod | file] [--] [arg] ...
...
2018-08-20 12:16:31 INFO ShutdownHookManager:54 - Shutdown hook called
2018-08-20 12:16:31 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-1f1e2815-bf26-4543-a524-42390ec7c6c1
所以看来我传递给myscript.py的标志是由ipython而不是实际的脚本拾取的。有什么办法解决吗？