java.lang.RuntimeException:找不到流“ /files/myScript.py”

时间:2018-07-02 19:59:05

标签: java apache-spark

在spark版本2.3.0中,我试图提交一个Python脚本以在launch()中使用org.apache.spark.launcher.SparkLauncher进行Spark,有时但并非总是如此,当我启动多个脚本时,我得到以下信息Spark日志中的错误:

2018-07-05 12:32:05 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 0
2018-07-05 12:32:05 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-07-05 12:32:05 INFO  Executor:54 - Fetching spark://myIP:49760/files/myScript.py with timestamp 1530786700005
2018-07-05 12:32:05 INFO  TransportClientFactory:267 - Successfully created connection to /myIP:49760 after 1 ms (0 ms spent in bootstraps)
2018-07-05 12:32:05 INFO  Utils:54 - Fetching spark://myIP:49760/files/myScript.py to /spark/tmp/spark-892ad2f3-107b-41ad-a432-d38f8810d4d1/executor-b5c8113f-c9e8-4402-a63c-5d35369d4c83/spark-90340267-1d33-4679-aa9f-2b6ab9eb6255/fetchFileTemp42232848203237515.tmp
2018-07-05 12:32:05 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.RuntimeException: Stream '/files/myScript.py' was not found.
     at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:242)
     at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:120)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
     at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
     at java.lang.Thread.run(Thread.java:745)

我检查了Spark日志,并在Spark将myScript文件复制到其tmp文件后抛出该日志。为什么找不到自己的文件?

0 个答案:

没有答案