Question

，我已经安装了 1）scala-2.10.3

2）spark-1.0.0 使用以下变量更改spark-env.sh

export SCALA_HOME=$HOME/scala-2.10.3

export SPARK_WORKER_MEMORY=16g

我可以看到Spark大师。

3）shark-0.9.1-bin-hadoop1

Changed shark-env.sh with below variables

export SHARK_MASTER_MEM=1g

SPARK_JAVA_OPTS=" -Dspark.local.dir=/tmp "

SPARK_JAVA_OPTS+="-Dspark.kryoserializer.buffer.mb=10 "

SPARK_JAVA_OPTS+="-verbose:gc -XX:-PrintGCDetails -XX:+PrintGCTimeStamps "

    export SPARK_JAVA_OPTS

    export HIVE_HOME=/usr/share/dse/hive

export HIVE_CONF_DIR="/etc/dse/hive"

export SPARK_HOME=/home/ubuntu/spark-1.0.0

export SPARK_MEM=16g

source $SPARK_HOME/conf/spark-env.sh

4）在DSE中，Hive版本是Hive 0.11 现有的Hive-site.xml是

    <configuration>

     <!-- Hive Execution Parameters -->
         <property>
                <name>hive.exec.mode.local.auto</name>
                <value>false</value>
             <description>Let hive determine whether to run in local mode automatically</description>
        </property>

        <property>
                <name>hive.metastore.warehouse.dir</name>
              <value>cfs:///user/hive/warehouse</value>
             <description>location of default database for the warehouse</description>
        </property>

        <property>
             <name>hive.hwi.war.file</name>
              <value>lib/hive-hwi.war</value>
                <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}</description>
          </property>

          <property>
                <name>hive.metastore.rawstore.impl</name>
                <value>com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStore</value>
                <description>Use the Apache Cassandra Hive RawStore implementation</description>
          </property>

         <property>
                <name>hadoop.bin.path</name>
                <value>${dse.bin}/dse hadoop</value>
          </property>

          <!-- Set this to true to enable auto-creation of Cassandra keyspaces as Hive Databases -->
          <property>
                <name>cassandra.autoCreateHiveSchema</name>
                <value>true</value>
          </property>
    </configuration>

5）运行Shark shell时出错：

 Unable to instantiate Org.apache.hadoop.hive.metastore.HiveMetaStoreClient

和

6）使用-skipRddReload运行shark shell时 - 我能够获得Shark shell但无法连接hive而无法执行任何命令。

 shark> DESCRIVE mykeyspace;

并收到错误消息：

FAILED: Error in metastore: java.lang.RuntimeException: Unable to instantiate org.apache.haddop.hive.metastore.HiveMataStoreClient.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.q1.exec.DDLTask.

请提供有关如何在Datastax enterprise（Cassandra）上配置spark / shark的详细信息。

在DSE上配置Spark / Shark时出错

0 个答案: