Question

我正在使用 Spark 2.3.1 和 Connector / J 5.1.47 。

我写了一个简单的程序来检查metastore的连接：

from pyspark.context import SparkContext
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf

conf = SparkConf()
conf.set("javax.jdo.option.ConnectionURL", "jdbc:mysql://localhost/my_metastore?createDatabaseIfNotExist=true&useSSL=false")
conf.set("javax.jdo.option.ConnectionDriverName", "com.mysql.jdbc.Driver")
conf.set("javax.jdo.option.ConnectionUserName", "root")
conf.set("javax.jdo.option.ConnectionPassword", "****")

spark = SparkSession.builder \
                .config(conf=conf) \
                .enableHiveSupport() \
                .getOrCreate()

spark.sql("SELECT NOW()").collect()

spark.stop()

令我惊讶的是，发现停止Spark会话后，Metastore连接仍然处于活动状态！

mysql> show processlist;
+------+------+-----------------+--------------+---------+------+----------+------------------+
| Id   | User | Host            | db           | Command | Time | State    | Info             |
+------+------+-----------------+--------------+---------+------+----------+------------------+
|    3 | lc   | localhost       | NULL         | Query   |    0 | starting | show processlist |
| 4342 | root | localhost:54368 | my_metastore | Sleep   |    5 |          | NULL             |
| 4343 | root | localhost:54369 | my_metastore | Sleep   |    5 |          | NULL             |
| 4346 | root | localhost:54372 | my_metastore | Sleep   |    5 |          | NULL             |
| 4347 | root | localhost:54373 | my_metastore | Sleep   |    5 |          | NULL             |
+------+------+-----------------+--------------+---------+------+----------+------------------+
5 rows in set (0.00 sec)

是否知道它是Spark还是Connector / J的错误？

停止Spark会话不会关闭Metastore MySQL连接

0 个答案: