我将内核配置为在jupyter实验室上运行pyspark客户端。
> jupyter kernelspec list
mykernel /home/mypath/jupyter/kernels/mykernel
内核配置文件(kernel.json
)有很多我想重用的参数
> cat kernel.json
{
"display_name": "mykernel",
"language": "python",
"argv": [
"/home/mypath/environments/python2/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env": {
"HADOOP_USER_NAME": "myusername",
"YARN_CONF_DIR": "/somepath/hadoop/conf",
"SPARK_HOME": "/somepath/spark/spark-2.0.0",
"PYTHONPATH": "/somepath/spark/spark-2.0.0/python/:/somepath/spark/spark-2.0.0/python/lib/py4j-0.10.1-src.zip",
"PYTHONSTARTUP": "/somepath/kernels/pyspark/start.py",
"PYSPARK_SUBMIT_ARGS": "--alotofargs --moreargs --moreargs"
}
}
是否可以通过命令行执行类似的操作?
jupyter mykernel run mypysparkmodule.py
非常感谢!