Question

要重现，请使用最简单的苏打水Python示例（https://github.com/h2oai/sparkling-water/blob/rel-2.2/py/examples/scripts/H2OContextInitDemo.py）：

from pysparkling import *
from pyspark.sql import SparkSession
import h2o

# Initiate SparkSession
spark = SparkSession.builder.appName("App name").getOrCreate()

# Initiate H2OContext
hc = H2OContext.getOrCreate(spark)

# Stop H2O and Spark services
h2o.shutdown(prompt=False)
spark.stop()

我已导出SPARK_HOME并指向Spark 2.2.0。我有MASTER =“local [4]”。

我已安装（其中包括）：

pyspark (2.2.0)
h2o-pysparkling-2.2 (2.2.2)
h2o (3.14.0.7)

现在，当我运行这个脚本时，我得到了（在Python 2.7下）：

H2O session _sid_9ee5 closed.
/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py:151: UserWarning: Stopping H2OContext. (Restarting H2O is not yet fully supported...) 
  warnings.warn("Stopping H2OContext. (Restarting H2O is not yet fully supported...) ")
11-02 17:37:43.710 10.0.1.62:54321       21323  Thread-28 INFO: Orderly shutdown:  Shutting down now.
11-02 17:37:43.719 10.0.1.62:54321       21323  Thread-29 INFO: Orderly shutdown:  Shutting down now.
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 883, in send_command
    response = connection.send_command(command)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 1040, in send_command
    "Error while receiving", e, proto.ERROR_ON_RECEIVE)
Py4JNetworkError: Error while receiving
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py", line 140, in <lambda>
    atexit.register(lambda: h2o_context.stop_with_jvm())
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py", line 147, in stop_with_jvm
    self.stop()
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py", line 153, in stop
    self._jhc.stop(False)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 1133, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/py4j/protocol.py", line 327, in get_return_value
    format(target_id, ".", name))
Py4JError: An error occurred while calling o32.stop
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py", line 140, in <lambda>
    atexit.register(lambda: h2o_context.stop_with_jvm())
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py", line 147, in stop_with_jvm
    self.stop()
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pysparkling/context.py", line 153, in stop
    self._jhc.stop(False)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/py4j/java_gateway.py", line 1133, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/user/.virtualenvs/sacred2/local/lib/python2.7/site-packages/py4j/protocol.py", line 327, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o32.stop

为什么我会得到那些追溯？脚本的返回码是0，也是在Python 3中，但是抛出了一些其他的回溯。如何清理它？

完整日志：https://gist.github.com/anonymous/163fba371b2a419c2171f4aff83a1ff7

Answer 1

这是一个错误，并已作为此JIRA https://0xdata.atlassian.net/browse/SW-569的一部分修复。这个错误修复将包含在下一个Sparkling Water版本中。

关闭Sparkling Water上下文时的奇怪回溯

1 个答案: