尽管停止了Spark上下文,但火花过程仍使用Derby的“ metastore_db”

时间:2019-03-14 14:07:16

标签: python apache-spark pyspark pyspark-sql

我试图在停止Spark上下文后删除“ metastore_db”,但是会引发以下错误:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'metastore_db\\db.lck'

代码如下:

import shutil
try:
    import pyspark
except:
    import findspark
    findspark.init()  

spark = pyspark.sql.SparkSession.builder.appName("Test_metastore_delete").enableHiveSupport().getOrCreate()
spark.stop()
shutil.rmtree("metastore_db")

PySpark文档说:

https://spark.apache.org/docs/2.3.1/api/python/pyspark.html#pyspark.SparkContext.stop

  • stop()关闭SparkContext。

任何帮助或建议,为什么以及如何解决此问题。

0 个答案:

没有答案