在DSE上配置Spark / Shark时出错

时间:2014-06-10 12:29:00

标签: cassandra hive datastax-enterprise shark-sql metastore

,我已经安装了 1)scala-2.10.3

2)spark-1.0.0     使用以下变量更改spark-env.sh

export SCALA_HOME=$HOME/scala-2.10.3

export SPARK_WORKER_MEMORY=16g

我可以看到Spark大师。

3)shark-0.9.1-bin-hadoop1

Changed shark-env.sh with below variables

export SHARK_MASTER_MEM=1g

SPARK_JAVA_OPTS=" -Dspark.local.dir=/tmp "

SPARK_JAVA_OPTS+="-Dspark.kryoserializer.buffer.mb=10 "

SPARK_JAVA_OPTS+="-verbose:gc -XX:-PrintGCDetails -XX:+PrintGCTimeStamps "

    export SPARK_JAVA_OPTS

    export HIVE_HOME=/usr/share/dse/hive

export HIVE_CONF_DIR="/etc/dse/hive"

export SPARK_HOME=/home/ubuntu/spark-1.0.0

export SPARK_MEM=16g

source $SPARK_HOME/conf/spark-env.sh

4)在DSE中,Hive版本是Hive 0.11     现有的Hive-site.xml是

    <configuration>

     <!-- Hive Execution Parameters -->
         <property>
                <name>hive.exec.mode.local.auto</name>
                <value>false</value>
             <description>Let hive determine whether to run in local mode automatically</description>
        </property>

        <property>
                <name>hive.metastore.warehouse.dir</name>
              <value>cfs:///user/hive/warehouse</value>
             <description>location of default database for the warehouse</description>
        </property>

        <property>
             <name>hive.hwi.war.file</name>
              <value>lib/hive-hwi.war</value>
                <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}</description>
          </property>

          <property>
                <name>hive.metastore.rawstore.impl</name>
                <value>com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStore</value>
                <description>Use the Apache Cassandra Hive RawStore implementation</description>
          </property>

         <property>
                <name>hadoop.bin.path</name>
                <value>${dse.bin}/dse hadoop</value>
          </property>

          <!-- Set this to true to enable auto-creation of Cassandra keyspaces as Hive Databases -->
          <property>
                <name>cassandra.autoCreateHiveSchema</name>
                <value>true</value>
          </property>
    </configuration>

5)运行Shark shell时出错:

 Unable to instantiate Org.apache.hadoop.hive.metastore.HiveMetaStoreClient

6)使用-skipRddReload运行shark shell时 - 我能够获得Shark shell但无法连接hive而无法执行任何命令。

 shark> DESCRIVE mykeyspace;

并收到错误消息:

FAILED: Error in metastore: java.lang.RuntimeException: Unable to instantiate org.apache.haddop.hive.metastore.HiveMataStoreClient.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.q1.exec.DDLTask.

请提供有关如何在Datastax enterprise(Cassandra)上配置spark / shark的详细信息。

0 个答案:

没有答案