如何使用来自远程源的jar在Kubernetes上运行apache spark 2.3.0时如何解决此错误

时间:2018-03-29 13:39:19

标签: apache-spark kubernetes

关注the instructions here我一直在尝试使用远程网址向minikube提交火花作业:

minikube start bin/spark-submit --master k8s://https://192.168.99.100:8443 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=<default-spark-k8s-image-build> --conf spark.kubernetes.namespace=spark <https://remote-location-with-spark-example-jar>

pod失败,当我描述它时,我收到错误 configmaps&#34; spark-pi-ad386ea0f7e4333dbd2a0ad705e94d66-init-config&#34;找不到 Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 50s default-scheduler Successfully assigned spark-pi-ad386ea0f7e4333dbd2a0ad705e94d66-driver to minikube Warning FailedMount 49s kubelet, minikube MountVolume.SetUp failed for volume "spark-init-properties" : **configmaps "spark-pi-ad386ea0f7e4333dbd2a0ad705e94d66-init-config" not found** Normal SuccessfulMountVolume 49s kubelet, minikube MountVolume.SetUp succeeded for volume "download-jars-volume" Normal SuccessfulMountVolume 49s kubelet, minikube MountVolume.SetUp succeeded for volume "download-files-volume" Normal SuccessfulMountVolume 49s kubelet, minikube MountVolume.SetUp succeeded for volume "default-token-4ghj8" Normal SuccessfulMountVolume 49s kubelet, minikube MountVolume.SetUp succeeded for volume "spark-init-properties" Normal Pulled 49s kubelet, minikube Container image "timg-spark/spark:latest" already present on machine Normal Created 49s kubelet, minikube Created container Normal Started 48s kubelet, minikube Started container Normal Pulled 43s kubelet, minikube Container image "timmeh/spark:latest" already present on machine Normal Created 43s kubelet, minikube Created container Normal Started 43s kubelet, minikube Started container

然而,在创建任何配置映射的文档中没有提及,并且因为在运行spark-submit之前不知道configmap的名称,所以我无法提前创建一个以获取更多信息

现在我的计划是通过将jar文件烘焙到spark docker图像来解决,但是如果有人知道为什么这会失败那就太棒了!

0 个答案:

没有答案