Question

我在CDH 5.12上有一个带有Spark 2.2的集群与RHEL，我正在尝试设置IPython以与pyspark2一起使用。我已经安装了IPython 5.x LTS（长期支持），但我无法让它工作。

到目前为止

yum -y update
yum install epel-release
yum -y install python-pip
yum groupinstall 'Development Tools'
yum install python-devel

pip install IPython==5.0 --user

但我无法让它发挥作用。任何人都知道我错过了什么？

Answer 1

pyspark启动脚本查找，

# Determine the Python executable to use for the driver:
if [[ -n "$IPYTHON_OPTS" || "$IPYTHON" == "1" ]]; then
  # If IPython options are specified, assume user wants to run IPython
  # (for backwards-compatibility)
  PYSPARK_DRIVER_PYTHON_OPTS="$PYSPARK_DRIVER_PYTHON_OPTS $IPYTHON_OPTS"
  PYSPARK_DRIVER_PYTHON="ipython"
elif [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
  PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"$DEFAULT_PYTHON"}"
fi

在~/.bashrc

中设置以下变量

echo "export PATH=$PATH:/path_to_downloaded_spark/spark-1.6.0/bin"
echo "export PYSPARK_DRIVER_PYTHON=ipython"
echo "export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

在CDH 5.12上使用Spark 2.2进行IPython安装

1 个答案: