Question

我目前正在使用Cloudera Express 5.5.0上的SAP HANA Vora迈出第一步。

Vora服务器启动并运行，我现在想使用Vora spark shell，但这就是我得到的：

export LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
export JAVA_HOME=/usr/java/default
export HADOOP_PARCEL_PATH=/opt/cloudera/parcels/CDH
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/usr/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PATH=$PATH:$SPARK_HOME/bin

SPARK_DIST_CLASSPATH=$SPARK_HOME/lib/spark-assembly.jar
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-hdfs/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-hdfs/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-mapreduce/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-mapreduce/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-yarn/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hadoop-yarn/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/hive/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/flume-ng/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/parquet/lib/*
SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:$HADOOP_PARCEL_PATH/lib/avro/lib/*
export SPARK_DIST_CLASSPATH

这就是我的环境：

springSecurityFilterChain

Answer 1

解决了它。

只需将Java从JDK6升级到JDK7即可。确保设置了以下环境变量（检查安装时的值）：

export LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
export JAVA_HOME=/usr/java/default
export HADOOP_PARCEL_PATH=/opt/cloudera/parcels/CDH
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/usr/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PATH=$PATH:$SPARK_HOME/bin

Answer 2

感谢您回答https://stackoverflow.com/users/1867854/michael-kunzmann。我发现JDK7已经安装在/usr/java/jdk1.7.0_67-cloudera目录下。我认为这是Cloudera Manager安装中的一个步骤。在撰写本文时，CDH 5.3,5.5,5.6的最低支持版本为1.7.0_55（例如见https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp）

我正在尝试使用stanard（非Vora）spark-shell并在CDH上遇到同样的问题。因此，标准火花壳也需要JDK7。 Vora spark-shell脚本..

$VORA_SPARK_HOME/bin/start-spark-shell.sh

...只是将Vora数据源jar添加为附加库。

仅供参考，这是Cloudera CDH上标准火花壳的一个例子..

~> cd /usr/java
/usr/java> ls -l
total 8
lrwxrwxrwx 1 lroot root   16 Dec 17  2015 default -> /usr/java/latest
drwxr-xr-x 9 lroot root 4096 Dec 17  2015 jdk1.6.0_31
drwxr-xr-x 8 lroot root 4096 Dec 17  2015 jdk1.7.0_67-cloudera
lrwxrwxrwx 1 lroot root   21 Dec 17  2015 latest -> /usr/java/jdk1.6.0_31
/usr/java> export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
/usr/java> export SPARK_HOME=/usr/lib/spark
/usr/java> cd $SPARK_HOME
/usr/lib/spark> ./bin/spark-shell

仅供参考我在HortonWorks上也有Vora。 Java 7已通过/ usr / bin / java sybolic链接在PATH上，这只是工作..

source /etc/vora/vora-env.sh
$VORA_SPARK_HOME/bin/start-spark-shell.sh

SAP HANA Vora shell无法找到org.apache.spark.launcher.Main

2 个答案: