Question

这些技术是全新的，感谢您对我遇到的问题有所了解。我已经下载了spark 2.4，并在ubuntu上构建了它（而不是建议的alpine），并尝试在我的测试kubernetes集群上运行。由于tini不可用，我将默认的entrypoint.sh最后一行更改为以下内容：exec "${CMD[@]}"（original file）

之所以这样做，是因为tini软件包在ubuntu中不可用。我不确定这是否行得通，但是我进行spark-submit时的最终结果是：

我的驱动程序窗格已创建
执行器pod不断终止并重新启动。

我在驱动程序日志中看到的全部是：

2019-02-19 19:47:52 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-02-19 19:47:52 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
2019-02-19 19:47:52 INFO  NettyBlockTransferService:54 - Server created on spark-pi-1550605656705-driver-svc.125874-poc.svc:7079
2019-02-19 19:47:52 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-02-19 19:47:53 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, spark-pi-1550605656705-driver-svc.125874-poc.svc, 7079, None)
2019-02-19 19:47:53 INFO  BlockManagerMasterEndpoint:54 - Registering block manager spark-pi-1550605656705-driver-svc.125874-poc.svc:7079 with 1007.8 MB RAM, BlockManagerId(driver, spark-pi-1550605656705-driver-svc.125874-poc.svc, 7079, None)
2019-02-19 19:47:53 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, spark-pi-1550605656705-driver-svc.125874-poc.svc, 7079, None)
2019-02-19 19:47:53 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, spark-pi-1550605656705-driver-svc.125874-poc.svc, 7079, None)
2019-02-19 19:47:53 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5fcfca62{/metrics/json,null,AVAILABLE,@Spark}
2019-02-19 19:47:56 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-02-19 19:47:56 INFO  BlockManagerMaster:54 - Removal of executor 1 requested
2019-02-19 19:47:56 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 1
2019-02-19 19:47:56 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 1 from BlockManagerMaster.
2019-02-19 19:47:58 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 2 from BlockManagerMaster.

我的猜测是这与我的入口点有关，但是我不确定如何调试和找出根本原因。

为记录起见，它在入口点脚本中与tini一起在高山上工作。

带有SparkPi示例的kubernetes上的Spark提交作业

0 个答案: