我一直在尝试将最新的spark 1.6与hive 0.14.0集成在一起。我只是想让Thrift服务器运行。我注意到,如果我不重写以下配置:(-conf spark.sql.hive.metastore.version = 0.14.0 --conf spark.sql.hive.metastore.jars = maven)调用start-thrifstserver时.sh spark脚本,然后由于hive 1.2.1之间的不兼容问题导致任何创建表查询失败,默认情况下由spark 1.6使用,而我的hive版本在prod中运行。但是,当我覆盖这两个配置时,那么当thrift服务器启动时,它不会像hive-site.xml中指定的那样连接到我的hive Metorore uri,而是厌倦了连接到derby数据库然后Thrift服务器无法正常启动。我错过了一些额外的覆盖吗?
请参阅下面的thrift服务器日志信息:
Loaded from file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar
java.vendor=Oracle Corporation
java.runtime.version=1.7.0_79-b15
user.dir=/
os.name=Linux
os.arch=amd64
os.version=2.6.32-504.23.4.el6.x86_64
derby.system.home=null
Database Class Loader started - derby.database.classpath=''
16/01/26 16:35:20 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (10.15.150.38:51475) with ID 20
16/01/26 16:35:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.15.150.38:52107 with 9.9 GB RAM, BlockManagerId(20, 10.15.150.38, 52107)
16/01/26 16:35:20 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (10.15.150.38:51479) with ID 48
16/01/26 16:35:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.15.150.38:47973 with 9.9 GB RAM, BlockManagerId(48, 10.15.150.38, 47973)
16/01/26 16:35:20 WARN Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@3cf4a477:an attempt to override final parameter: mapreduce.reduce.speculative; Ignoring.
16/01/26 16:35:20 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/01/26 16:35:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/01/26 16:35:22 INFO ObjectStore: Initialized ObjectStore
16/01/26 16:35:22 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/01/26 16:35:22 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/01/26 16:35:22 INFO HiveMetaStore: Added admin role in metastore
16/01/26 16:35:22 INFO HiveMetaStore: Added public role in metastore
16/01/26 16:35:22 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/01/26 16:35:22 INFO HiveMetaStore: 0: get_all_databases
16/01/26 16:35:22 INFO audit: ugi=hive ip=unknown-ip-addr cmd=get_all_databases
16/01/26 16:35:22 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/01/26 16:35:22 INFO audit: ugi=hive ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/01/26 16:35:22 INFO SessionState: Created local directory: /tmp/06895c7e-e26c-42b7-b100-4222d0356b6b_resources
16/01/26 16:35:22 INFO SessionState: Created HDFS directory: /tmp/hive/hive/06895c7e-e26c-42b7-b100-4222d0356b6b
16/01/26 16:35:22 INFO SessionState: Created local directory: /tmp/hive/06895c7e-e26c-42b7-b100-4222d0356b6b
16/01/26 16:35:23 INFO SessionState: Created HDFS directory: /tmp/hive/hive/06895c7e-e26c-42b7-b100-4222d0356b6b/_tmp_space.db
16/01/26 16:35:23 WARN Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@37f031a:an attempt to override final parameter: mapreduce.reduce.speculative; Ignoring.
16/01/26 16:35:23 INFO HiveContext: default warehouse location is /user/hive/warehouse
16/01/26 16:35:23 INFO HiveContext: Initializing HiveMetastoreConnection version 0.14.0 using maven.
Ivy Default Cache set to: /home/hive/.ivy2/cache
The jars for the packages stored in: /home/hive/.ivy2/jars
http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1
:: loading settings :: url = jar:file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.calcite#calcite-core added as a dependency
org.apache.calcite#calcite-avatica added as a dependency
org.apache.hive#hive-metastore added as a dependency
org.apache.hive#hive-exec added as a dependency
org.apache.hive#hive-common added as a dependency
org.apache.hive#hive-serde added as a dependency
com.google.guava#guava added as a dependency
org.apache.hadoop#hadoop-client added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]