此问题类似于：sc is not created automatically in notebook和IBM Bluemix sc not defined。
我已重新启动内核，但错误仍然存在。
我已将日志文件转储为：
# dump the latest kernel log
! cat $(ls -1 $HOME/logs/notebook/*pyspark* | sort -r | head -1)
这似乎是一个内部错误：
...
...
17/04/30 13:54:05 INFO spark.util.Utils: Successfully started service 'sparkDriver' on port 42341.
17/04/30 13:54:05 INFO apache.spark.SparkEnv: The address of rpcenv is :10.143.133.23:42341
17/04/30 13:54:05 INFO event.slf4j.Slf4jLogger: Slf4jLogger started
17/04/30 13:54:05 INFO Remoting: Starting remoting
17/04/30 13:54:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.143.133.23:44509]
17/04/30 13:54:06 INFO spark.util.Utils: Successfully started service 'sparkDriverActorSystem' on port 44509.
17/04/30 13:54:06 INFO apache.spark.SparkEnv: Registering MapOutputTracker 17/04/30 13:54:06 INFO apache.spark.SparkEnv: Registering BlockManagerMaster 17/04/30 13:54:06 INFO spark.storage.DiskBlockManager: Created local directory at /tmp/spark-160-ego-master/work/blockmgr-084c6902-4534-4669-9c37-4cee8c135574 17/04/30 13:54:06 INFO spark.storage.MemoryStore: MemoryStore started with capacity 909.0 MB 17/04/30 13:54:06 INFO apache.spark.SparkEnv: Registering OutputCommitCoordinator
17/04/30 13:54:06 INFO spark.util.EGOSparkDockerConfig: Docker not enabled
17/04/30 13:54:06 INFO cluster.ego.EGOFineGrainedSchedulerBackend: setting reserve=0, priority=1, limit=2147483647, master=spark://yp-spark-dal09-env5-0018:7082
17/04/30 13:54:06 INFO client.ego.EGOAppClient$ClientEndpoint: Connecting to master spark://yp-spark-dal09-env5-0018:7082...
17/04/30 13:54:26 INFO client.ego.EGOAppClient$ClientEndpoint: Connecting to master spark://yp-spark-dal09-env5-0018:7082...
17/04/30 13:54:46 INFO client.ego.EGOAppClient$ClientEndpoint: Connecting to master spark://yp-spark-dal09-env5-0018:7082...
17/04/30 13:54:46 ERROR spark.util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e6797d79 rejected from java.util.concurrent.ThreadPoolExecutor@b63fafd[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2058)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:834)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1380)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at org.apache.spark.deploy.client.ego.EGOAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(EGOAppClient.scala:125)
at org.apache.spark.deploy.client.ego.EGOAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(EGOAppClient.scala:124)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.deploy.client.ego.EGOAppClient$ClientEndpoint.tryRegisterAllMasters(EGOAppClient.scala:124)
at org.apache.spark.deploy.client.ego.EGOAppClient$ClientEndpoint.org$apache$spark$deploy$client$ego$EGOAppClient$ClientEndpoint$$registerWithMaster(EGOAppClient.scala:150)
at org.apache.spark.deploy.client.ego.EGOAppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(EGOAppClient.scala:163)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1245)
at org.apache.spark.deploy.client.ego.EGOAppClient$ClientEndpoint$$anon$2.run(EGOAppClient.scala:153)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)
17/04/30 13:54:46 INFO spark.storage.DiskBlockManager: Shutdown hook called
17/04/30 13:54:46 INFO spark.util.ShutdownHookManager: Shutdown hook called
17/04/30 13:54:46 INFO spark.util.ShutdownHookManager: Deleting directory /tmp/spark-160-ego-master/work/spark-4e64a5fb-57d4-4ef4-b313-93f6468ca270/userFiles-9e971358-d7e7-457d-b815-4d3d8b45c0e3
17/04/30 13:54:46 INFO spark.util.ShutdownHookManager: Deleting directory /tmp/spark-160-ego-master/work/spark-4e64a5fb-57d4-4ef4-b313-93f6468ca270
ERROR:py4j.java_gateway:Error while sending or receiving.
Traceback (most recent call last):
File "/usr/local/src/spark160master/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command
raise Py4JError("Answer from Java side is empty")
Py4JError: Answer from Java side is empty
ERROR:py4j.java_gateway:Error while sending or receiving.
Traceback (most recent call last):
File "/usr/local/src/spark160master/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command
raise Py4JError("Answer from Java side is empty")
Py4JError: Answer from Java side is empty
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server
Traceback (most recent call last):
File "/usr/local/src/spark160master/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 690, in start
self.socket.connect((self.address, self.port))
File "/usr/local/src/bluemix_jupyter_bundle.v41/notebook/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /usr/local/src/spark160master/spark/python/pyspark/shell.py:
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server
Traceback (most recent call last):
File "/usr/local/src/spark160master/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 690, in start
self.socket.connect((self.address, self.port))
File "/usr/local/src/bluemix_jupyter_bundle.v41/notebook/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server
Traceback (most recent call last):
File "/usr/local/src/spark160master/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 690, in start
self.socket.connect((self.address, self.port))
File "/usr/local/src/bluemix_jupyter_bundle.v41/notebook/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused