Spark1.6和Hive 0.14集成问题

时间:2016-01-27 01:17:15

标签: apache-spark hive thrift

我一直在尝试将最新的spark 1.6与hive 0.14.0集成在一起。我只是想让Thrift服务器运行。我注意到,如果我不重写以下配置:(-conf spark.sql.hive.metastore.version = 0.14.0 --conf spark.sql.hive.metastore.jars = maven)调用start-thrifstserver时.sh spark脚本,然后由于hive 1.2.1之间的不兼容问题导致任何创建表查询失败,默认情况下由spark 1.6使用,而我的hive版本在prod中运行。但是,当我覆盖这两个配置时,那么当thrift服务器启动时,它不会像hive-site.xml中指定的那样连接到我的hive Metorore uri,而是厌倦了连接到derby数据库然后Thrift服务器无法正常启动。我错过了一些额外的覆盖吗?

请参阅下面的thrift服务器日志信息:

Loaded from file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar 
java.vendor=Oracle Corporation 
java.runtime.version=1.7.0_79-b15 
user.dir=/ 
os.name=Linux 
os.arch=amd64 
os.version=2.6.32-504.23.4.el6.x86_64 
derby.system.home=null 
Database Class Loader started - derby.database.classpath='' 
16/01/26 16:35:20 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (10.15.150.38:51475) with ID 20 
16/01/26 16:35:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.15.150.38:52107 with 9.9 GB RAM, BlockManagerId(20, 10.15.150.38, 52107) 
16/01/26 16:35:20 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (10.15.150.38:51479) with ID 48 
16/01/26 16:35:20 INFO BlockManagerMasterEndpoint: Registering block manager 10.15.150.38:47973 with 9.9 GB RAM, BlockManagerId(48, 10.15.150.38, 47973) 
16/01/26 16:35:20 WARN Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@3cf4a477:an attempt to override final parameter: mapreduce.reduce.speculative;  Ignoring. 
16/01/26 16:35:20 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 
16/01/26 16:35:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 
16/01/26 16:35:21 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 
16/01/26 16:35:22 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 
16/01/26 16:35:22 INFO ObjectStore: Initialized ObjectStore 
16/01/26 16:35:22 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 
16/01/26 16:35:22 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 
16/01/26 16:35:22 INFO HiveMetaStore: Added admin role in metastore 
16/01/26 16:35:22 INFO HiveMetaStore: Added public role in metastore 
16/01/26 16:35:22 INFO HiveMetaStore: No user is added in admin role, since config is empty 
16/01/26 16:35:22 INFO HiveMetaStore: 0: get_all_databases 
16/01/26 16:35:22 INFO audit: ugi=hive  ip=unknown-ip-addr      cmd=get_all_databases 
16/01/26 16:35:22 INFO HiveMetaStore: 0: get_functions: db=default pat=* 
16/01/26 16:35:22 INFO audit: ugi=hive  ip=unknown-ip-addr      cmd=get_functions: db=default pat=* 
16/01/26 16:35:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table. 
16/01/26 16:35:22 INFO SessionState: Created local directory: /tmp/06895c7e-e26c-42b7-b100-4222d0356b6b_resources 
16/01/26 16:35:22 INFO SessionState: Created HDFS directory: /tmp/hive/hive/06895c7e-e26c-42b7-b100-4222d0356b6b 
16/01/26 16:35:22 INFO SessionState: Created local directory: /tmp/hive/06895c7e-e26c-42b7-b100-4222d0356b6b 
16/01/26 16:35:23 INFO SessionState: Created HDFS directory: /tmp/hive/hive/06895c7e-e26c-42b7-b100-4222d0356b6b/_tmp_space.db 
16/01/26 16:35:23 WARN Configuration: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@37f031a:an attempt to override final parameter: mapreduce.reduce.speculative;  Ignoring. 
16/01/26 16:35:23 INFO HiveContext: default warehouse location is /user/hive/warehouse 
16/01/26 16:35:23 INFO HiveContext: Initializing HiveMetastoreConnection version 0.14.0 using maven. 
Ivy Default Cache set to: /home/hive/.ivy2/cache 
The jars for the packages stored in: /home/hive/.ivy2/jars 
http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1 
:: loading settings :: url = jar:file:/usr/lib/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml 
org.apache.calcite#calcite-core added as a dependency 
org.apache.calcite#calcite-avatica added as a dependency 
org.apache.hive#hive-metastore added as a dependency 
org.apache.hive#hive-exec added as a dependency 
org.apache.hive#hive-common added as a dependency 
org.apache.hive#hive-serde added as a dependency 
com.google.guava#guava added as a dependency 
org.apache.hadoop#hadoop-client added as a dependency 
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 
        confs: [default]

0 个答案:

没有答案