Apache apive 2.1.1 on apache spark 2.0

时间:2018-02-08 13:34:18

标签: hadoop hive apache-spark-2.0 beeline

在hive中连接我的beeline时,无法创建spark客户端

select count(*) from student;
Query ID = hadoop_20180208184224_f86b5aeb-f27b-4156-bd77-0aab54c0ec67
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>

执行spark任务失败,异常

  

org.apache.hadoop.hive.ql.metadata.HiveException(无法创建spark客户端。)

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=1)

在独立群集模式下安装了spark prebuilt 2.0

我的hive-site.xml - 也放在spark / conf中并删除了hdfs路径中的hive jar

更新了hive-site.xml

<property>
<name>hive.execution.engine</name>
<value>spark</value>
<description>
</property>
   <property>
    <name>hive.metastore.uris</name>
    <value/>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>
    <property>
    <name>spark.master</name>
    <value>yarn</value>
    <description>Spark Master URL</description>
    </property>
    <property>
    <name>spark.eventLog.enabled</name>
    <value>true</value>
    <description>Spark Event Log</description>
    </property>
    <property>
    <name>spark.eventLog.dir</name>
    <value>hdfs://10.196.220.131:9000/user/spark/eventLogging</value>
    <description>Spark event log folder</description>
    </property>
    <property>
    <name>spark.executor.memory</name>
    <value>512m</value>
    <description>Spark executor memory</description>
    </property>
    <property>
    <name>spark.serializer</name>
    <value>org.apache.spark.serializer.KryoSerializer</value>
    <description>Spark serializer</description>
    </property>
    <property>
    <name>spark.yarn.jars</name>
    <value>hdfs://10.196.220.131:9000/user/spark/spark-jars/*</value>
    </property>
    <property>
    <name>spark.submit.deployMode</name>
    <value>cluster</value>
    <description>Spark Master URL</description>
    </property>

使用Mysqldatabase连接 - 本地Metastore模式

   a) ConnectionURL
                    <name>javax.jdo.option.ConnectionURL</name>
                    <value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
                 b) ConnectionUserName
                    <name>javax.jdo.option.ConnectionUserName</name>
                    <value>hiveuser</value> 
                 c) ConnectionPassword
                    <name>javax.jdo.option.ConnectionPassword</name>
                    <value>xxxx</value>
                 d) ConnectionDriver
                    <name>javax.jdo.option.ConnectionDriverName</name>
                    <value>com.mysql.jdbc.Driver</value>

我的yarn-site.xml

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>40960</value>  
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>  
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>8192</value>  
</property>

我的RM网络用户界面日志

18/02/09 18:08:39 INFO spark.SecurityManager: Changing view acls groups to: 
18/02/09 18:08:39 INFO spark.SecurityManager: Changing modify acls groups to: 
18/02/09 18:08:39 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/02/09 18:08:39 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/JavaSparkListener
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:622)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:379)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:749)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:747)
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.JavaSparkListener
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 29 more
18/02/09 18:08:39 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)
18/02/09 18:08:39 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Shutdown hook called before final status was reported.)
18/02/09 18:08:39 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://10.196.220.131:9000/user/hadoop/.sparkStaging/application_1518178947017_0002
18/02/09 18:08:39 INFO util.ShutdownHookManager: Shutdown hook called

0 个答案:

没有答案