Question

连接Spark（针对Hadoop 2.6.0构建的1.4.0）和hive（版本1.1.0）表时，由于德比版本不匹配，异常即将发生。
两种方式都有（spark和hive））可以使用相同的德比版本？ ......还有其他方法来解决这个问题吗？

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@e0d0b81, see the next exception for details.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
    ... 88 more
Caused by: java.sql.SQLException: Database at /home/saurabh/Softwares/hive1.1/bin/metastore_db has an incompatible format with the current version of the software.  The database was created by or upgraded by version 10.11.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
    ... 85 more
Caused by: ERROR XSLAN: Database at /home/saurabh/Softwares/hive1.1/bin/metastore_db has an incompatible format with the current version of the software.  The database was created by or upgraded by version 10.11.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown Source)
    at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
    at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
    at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown Source)
    at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown Source)
    at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
    at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
    at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
    at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
    at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
    at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
    at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
    at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
    ... 85 more
15/07/01 17:29:49 INFO SparkUI: Stopped Spark web UI at http://192.168.17.155:4040
15/07/01 17:29:49 INFO DAGScheduler: Stopping DAGScheduler

Answer 1

这就是我解决它的方法：

rm -rf "${APPDATA//\\//}/../Local/Temp/spark-"*
rm -rf /c/tmp/hive
rm -rf ./metastore_db

我从git bash中删除了：

本地火花目录（非强制性但很好）
临时配置单元目录（必填）
metastore_db（必须删除metastore_db / * .loc我删除了整件事以确定）

我不确定它是否完全适用于您的情况，但它重置了derby数据库以便我可以使用它。

Answer 2

我的火花正在使用derby-10.12.1.1.jar，蜂巢正在使用derby-10.10.2.0.jar，因此转到/usr/local/spark/jars/并复制了derby-10.12.1.1.jar并粘贴到/usr/local/hive/lib

以上步骤解决了我的问题。

Answer 3

这是由于Spark和Hive之间的德比版本不兼容。在我的情况下，spark的德比值为10.12，而蜂巢的值为10.14。我从/ hive / lib目录将derby-10.14.1.0.jar复制到spark / jars目录，问题得到解决。

Spark和Hive之间的Derby版本不匹配：无法实例化org.apache.hadoop.hive.metastore.HiveMetaStoreClient

3 个答案: