Question

我正在尝试在k8s上使用spark。

发射的迷你库

minikube --memory 8192 --cpus 2 start

并构建spark master版本（新获取）并构建docker image并推送至docker hub并发出命令。

$SPARK_HOME/bin/spark-submit \
  --master k8s://192.168.99.100:8443 \
  --deploy-mode cluster --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=ruseel/spark:testing \ 
  local:///tmp/spark-examples_2.11-2.4.0-SNAPSHOT-shaded.jar

但是豆荚日志说

...
+ case "$SPARK_K8S_CMD" in
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_JAVA_OPTS[@]}" -cp "$SPARK_CLASSPATH" -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $SPARK_DRIVER_ARGS)
+ exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -cp ':/opt/spark/jars/*' -Xms -Xmx -Dspark.driver.bindAddress=172.17.0.4
Invalid initial heap size: -Xms
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

如何运行此命令？

Answer 1

Spark管理员的新entrypoint.sh未使用$SPARK_DRIVER_MEMORY。它似乎已在this commit中删除。所以这个错误对我来说不再存在。

k8s上产生火花-错误“无效的初始堆大小：-Xms”

1 个答案: