Spark-Submit，DataNucleus和HiveMetaStoreClient

时间：2016-05-04 10:09:07

标签： maven hadoop hive apache-spark-sql

我在Spark程序上有一个Hive，在Hive中创建一个表。

  • Hadoop 2.6.0.3.0.0.0-249
  • Spark 1.2.1
  • Hive 0.14

代码：

package tld.organization.artifactId

import org.apache.spark.SparkContext
import org.apache.spark.sql.hive._
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.SparkConf

object BuildTable {
    def main(args: Array[String]) {
        val sparkConf = new SparkConf().setAppName("ProcessFromSpark")
        sparkConf.setMaster("yarn-cluster");
        val sc = new SparkContext(sparkConf);
        val hiveContext = new HiveContext(sc)
        hiveContext.sql("CREATE TABLE IF NOT EXISTS DB.TEST_TABLE (ID VARCHAR(255))");
    }
}

在Eclipse上使用Maven编译，使用以下pom.xml 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>tld.organization</groupId>
    <artifactId>artifactId</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>

            <!-- for fatjar -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>assemble-all</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>fully.qualified.MainClass</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <version>2.6</version>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>spark-avro_2.10</artifactId>
            <version>0.2.0</version>
        </dependency>
        <dependency>
            <groupId>commons-dbutils</groupId>
            <artifactId>commons-dbutils</artifactId>
            <version>1.6</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.10</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>
</project>

使用以下命令

在群集上提交包本身 
spark-submit --master yarn-cluster --queue poc --num-executors 1 --executor-cores 1 --driver-memory 5G --executor-memory 5G --class tld.organization.artifactId.BuildTable hdfs://clustername/user/me/artifactId-0.0.1-SNAPSHOT-jar-with-dependencies.jar --jars spark-assembly-1.2.1.3.0.0.0-249-hadoop2.6.0.3.0.0.0-249.jar,datanucleus-api-jdo-3.2.6.jar,datanucleus-core-3.2.10.jar,datanucleus-rdbms-3.2.9.jar --files hive-site.xml

代码一直有效，直到Hive语句然后失败。 spark-submit输出：

Spark assembly has been built with Hive, including Datanucleus jars on classpath

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
INFO impl.TimelineClientImpl: Timeline service address: http://srv3:8188/ws/v1/timeline/
INFO yarn.Client: Requesting a new application from cluster with 76 NodeManagers
INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (78848 MB per container)
INFO yarn.Client: Will allocate AM container, with 5504 MB memory including 384 MB overhead
INFO yarn.Client: Setting up container launch context for our AM
INFO yarn.Client: Preparing resources for our AM container
WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1507269 for user on ha-hdfs:clustername
INFO yarn.Client: Setting up the launch environment for our AM container
WARN yarn.Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
INFO spark.SecurityManager: Changing view acls to: me
INFO spark.SecurityManager: Changing modify acls to: me
INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(me); users with modify permissions: Set(me)
INFO yarn.Client: Submitting application 1055 to ResourceManager
INFO impl.YarnClientImpl: Submitted application app_1461_1055
INFO yarn.Client: Application report for app_1461_1055 (state: ACCEPTED)
INFO yarn.Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: poc
         start time: 1462351844276
         final status: UNDEFINED
         tracking URL: http://servername:port/proxy/app_1461_1055/
         user: me
INFO yarn.Client: Application report for app_1461_1055 (state: ACCEPTED)
INFO yarn.Client: Application report for app_1461_1055 (state: RUNNING)
INFO yarn.Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: srv
         ApplicationMaster RPC port: 0
         queue: poc
         start time: 1462351844276
         final status: UNDEFINED
         tracking URL: http://servername:port/proxy/app_1461948861046_1055/
         user: me
INFO yarn.Client: Application report for app_1461948861046_1055 (state: RUNNING)
INFO yarn.Client: Application report for app_1461948861046_1055 (state: ACCEPTED)
INFO yarn.Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: poc
         start time: 1462351844276
         final status: UNDEFINED
         tracking URL: http://servername:port/proxy/app_1461_1055/
         user: me
INFO yarn.Client: Application report for app_1461_1055 (state: ACCEPTED)
INFO yarn.Client: Application report for app_1461_1055 (state: RUNNING)
INFO yarn.Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: srv2
         ApplicationMaster RPC port: 0
         queue: poc
         start time: 1462351844276
         final status: UNDEFINED
         tracking URL: http://servername:port/proxy/app_1461_1055/
         user: me
INFO yarn.Client: Application report for app_1461_1055 (state: RUNNING)
INFO yarn.Client: Application report for app_1461_1055 (state: FINISHED)
INFO yarn.Client:
         client token: N/A
         diagnostics: User class threw exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
         ApplicationMaster host: srv2
         ApplicationMaster RPC port: 0
         queue: poc
         start time: 1462351844276
         final status: FAILED
         tracking URL: http://servername:port/proxy/app_1461_1055/
         user: me
Error: application failed with exception
org.apache.spark.SparkException: Application finished with failed status
        at org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:522)
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我在submit命令中尝试了各种变体，包括hive-site.xml或将其排除（与jar相同），但没有任何效果。

我做错了什么？

编辑：删除了yarn logs输出，达到了30000个字符的限制。

hive-site.xml如下所示：

<configuration>
<property>
  <name>ambari.hive.db.schema.name</name>
  <value>hive</value>
</property>

<property>
  <name>datanucleus.cache.level2.type</name>
  <value>none</value>
</property>

<property>
  <name>hive.auto.convert.join</name>
  <value>true</value>
</property>

<property>
  <name>hive.auto.convert.join.noconditionaltask</name>
  <value>true</value>
</property>

<property>
  <name>hive.auto.convert.join.noconditionaltask.size</name>
  <value>1251999744</value>
</property>

<property>
  <name>hive.auto.convert.sortmerge.join</name>
  <value>true</value>
</property>

<property>
  <name>hive.auto.convert.sortmerge.join</name>
  <value>true</value>
</property>

<property>
  <name>hive.auto.convert.sortmerge.join.noconditionaltask</name>
  <value>true</value>
</property>

<property>
  <name>hive.auto.convert.sortmerge.join.to.mapjoin</name>
  <value>false</value>
</property>

<property>
  <name>hive.cbo.enable</name>
  <value>true</value>
</property>

<property>
  <name>hive.cli.print.header</name>
  <value>false</value>
</property>

<property>
  <name>hive.cluster.delegation.token.store.class</name>
  <value>org.apache.hadoop.hive.thrift.ZooKeeperTokenStore</value>
</property>

<property>
  <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
  <value>srv1:2181,srv2:2181,srv3:2181</value>
</property>

<property>
  <name>hive.cluster.delegation.token.store.zookeeper.znode</name>
  <value>/hive/cluster/delegation</value>
</property>

<property>
  <name>hive.compactor.abortedtxn.threshold</name>
  <value>1000</value>
</property>

<property>
  <name>hive.compactor.check.interval</name>
  <value>300L</value>
</property>

<property>
  <name>hive.compactor.delta.num.threshold</name>
  <value>10</value>
</property>

<property>
  <name>hive.compactor.delta.pct.threshold</name>
  <value>0.1f</value>
</property>

<property>
  <name>hive.compactor.initiator.on</name>
  <value>false</value>
</property>

<property>
  <name>hive.compactor.worker.threads</name>
  <value>0</value>
</property>

<property>
  <name>hive.compactor.worker.timeout</name>
  <value>86400L</value>
</property>

<property>
  <name>hive.compute.query.using.stats</name>
  <value>true</value>
</property>

<property>
  <name>hive.conf.restricted.list</name>
  <value>hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role</value>
</property>

<property>
  <name>hive.conf.validation</name>
  <value>false</value>
</property>

<property>
  <name>hive.convert.join.bucket.mapjoin.tez</name>
  <value>false</value>
</property>

<property>
  <name>hive.enforce.bucketing</name>
  <value>true</value>
</property>

<property>
  <name>hive.enforce.sorting</name>
  <value>true</value>
</property>

<property>
  <name>hive.enforce.sortmergebucketmapjoin</name>
  <value>true</value>
</property>

<property>
  <name>hive.exec.compress.intermediate</name>
  <value>false</value>
</property>

<property>
  <name>hive.exec.compress.output</name>
  <value>false</value>
</property>

<property>
  <name>hive.exec.dynamic.partition</name>
  <value>true</value>
</property>

<property>
  <name>hive.exec.dynamic.partition.mode</name>
  <value>nonstrict</value>
</property>

<property>
  <name>hive.exec.failure.hooks</name>
  <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
</property>

<property>
  <name>hive.exec.max.created.files</name>
  <value>100000</value>
</property>

<property>
  <name>hive.exec.max.dynamic.partitions</name>
  <value>5000</value>
</property>

<property>
  <name>hive.exec.max.dynamic.partitions.pernode</name>
 <value>2000</value>
</property>

<property>
  <name>hive.exec.orc.compression.strategy</name>
  <value>SPEED</value>
</property>

<property>
  <name>hive.exec.orc.default.compress</name>
  <value>ZLIB</value>
</property>

<property>
  <name>hive.exec.orc.default.stripe.size</name>
  <value>67108864</value>
</property>

<property>
  <name>hive.exec.parallel</name>
  <value>false</value>
</property>

<property>
  <name>hive.exec.parallel.thread.number</name>
  <value>8</value>
</property>

<property>
  <name>hive.exec.post.hooks</name>
  <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
</property>

<property>
  <name>hive.exec.pre.hooks</name>
  <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
</property>

<property>
  <name>hive.exec.reducers.bytes.per.reducer</name>
  <value>67108864</value>
</property>

<property>
  <name>hive.exec.reducers.max</name>
  <value>1009</value>
</property>

<property>
  <name>hive.exec.scratchdir</name>
  <value>/tmp/hive</value>
</property>

<property>
  <name>hive.exec.submit.local.task.via.child</name>
  <value>true</value>
</property>

<property>
  <name>hive.exec.submitviachild</name>
  <value>false</value>
</property>

<property>
  <name>hive.execution.engine</name>
  <value>tez</value>
</property>

<property>
  <name>hive.fetch.task.aggr</name>
  <value>false</value>
</property>

<property>
  <name>hive.fetch.task.conversion</name>
  <value>more</value>
</property>

<property>
  <name>hive.fetch.task.conversion.threshold</name>
  <value>1073741824</value>
</property>

<property>
  <name>hive.heapsize</name>
  <value>2048</value>
</property>

<property>
  <name>hive.limit.optimize.enable</name>
  <value>true</value>
</property>

<property>
  <name>hive.limit.pushdown.memory.usage</name>
  <value>0.04</value>
</property>

<property>
  <name>hive.map.aggr</name>
  <value>true</value>
</property>

<property>
  <name>hive.map.aggr.hash.force.flush.memory.threshold</name>
  <value>0.9</value>
</property>

<property>
  <name>hive.map.aggr.hash.min.reduction</name>
  <value>0.5</value>
</property>

<property>
  <name>hive.map.aggr.hash.percentmemory</name>
  <value>0.5</value>
</property>

<property>
  <name>hive.mapjoin.bucket.cache.size</name>
  <value>10000</value>
</property>

<property>
  <name>hive.mapjoin.optimized.hashtable</name>
  <value>true</value>
</property>

<property>
  <name>hive.mapred.reduce.tasks.speculative.execution</name>
  <value>false</value>
</property>

<property>
  <name>hive.merge.mapfiles</name>
  <value>true</value>
</property>

<property>
  <name>hive.merge.mapredfiles</name>
  <value>false</value>
</property>

<property>
  <name>hive.merge.orcfile.stripe.level</name>
  <value>true</value>
</property>

<property>
  <name>hive.merge.rcfile.block.level</name>
  <value>true</value>
</property>

<property>
  <name>hive.merge.size.per.task</name>
  <value>256000000</value>
</property>

<property>
  <name>hive.merge.smallfiles.avgsize</name>
  <value>16000000</value>
</property>

<property>
  <name>hive.merge.tezfiles</name>
  <value>false</value>
</property>

<property>
  <name>hive.metastore.authorization.storage.checks</name>
  <value>false</value>
</property>

<property>
  <name>hive.metastore.cache.pinobjtypes</name>
  <value>Table,Database,Type,FieldSchema,Order</value>
</property>

<property>
  <name>hive.metastore.client.connect.retry.delay</name>
  <value>5s</value>
</property>

<property>
  <name>hive.metastore.client.socket.timeout</name>
  <value>1800s</value>
</property>

<property>
  <name>hive.metastore.connect.retries</name>
  <value>24</value>
</property>

<property>
  <name>hive.metastore.execute.setugi</name>
  <value>true</value>
</property>

<property>
  <name>hive.metastore.failure.retries</name>
  <value>24</value>
</property>

<property>
  <name>hive.metastore.kerberos.keytab.file</name>
  <value>/etc/security/keytabs/hive.service.keytab</value>
</property>

<property>
  <name>hive.metastore.kerberos.principal</name>
  <value>hive/_HOST@HADOOP.HPC.MUC</value>
</property>

<property>
  <name>hive.metastore.pre.event.listeners</name>
  <value>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</value>
</property>

<property>
  <name>hive.metastore.sasl.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.metastore.server.max.threads</name>
  <value>100000</value>
</property>

<property>
  <name>hive.metastore.uris</name>
  <value>thrift://srv2:9083</value>
</property>

<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/apps/hive/warehouse</value>
</property>

<property>
  <name>hive.optimize.bucketmapjoin</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.bucketmapjoin.sortedmerge</name>
  <value>false</value>
</property>

<property>
  <name>hive.optimize.constant.propagation</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.index.filter</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.mapjoin.mapreduce</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.metadataonly</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.null.scan</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.reducededuplication</name>
  <value>true</value>
</property>

<property>
  <name>hive.optimize.reducededuplication.min.reducer</name>
  <value>4</value>
</property>

<property>
  <name>hive.optimize.sort.dynamic.partition</name>
  <value>false</value>
</property>

<property>
  <name>hive.orc.compute.splits.num.threads</name>
  <value>10</value>
</property>

<property>
  <name>hive.orc.splits.include.file.footer</name>
  <value>false</value>
</property>

<property>
  <name>hive.prewarm.enabled</name>
  <value>false</value>
</property>

<property>
  <name>hive.prewarm.numcontainers</name>
  <value>10</value>
</property>

<property>
  <name>hive.security.authenticator.manager</name>
  <value>org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator</value>
</property>

<property>
  <name>hive.security.authorization.enabled</name>
  <value>false</value>
</property>

<property>
  <name>hive.security.authorization.manager</name>

org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider     

<property>
  <name>hive.security.metastore.authenticator.manager</name>
  <value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value>
</property>

<property>
  <name>hive.security.metastore.authorization.auth.reads</name>
  <value>true</value>
</property>

<property>
  <name>hive.security.metastore.authorization.manager</name>
  <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
</property>

<property>
  <name>hive.srv2.allow.user.substitution</name>
  <value>true</value>
</property>

<property>
  <name>hive.srv2.authentication</name>
  <value>KERBEROS</value>
</property>

<property>
  <name>hive.srv2.authentication.kerberos.keytab</name>
  <value>/etc/security/keytabs/hive.service.keytab</value>
</property>

<property>
  <name>hive.srv2.authentication.kerberos.principal</name>
  <value>hive/_HOST@HADOOP.HPC.MUC</value>
</property>

<property>
  <name>hive.srv2.authentication.spnego.keytab</name>
  <value>/etc/security/keytabs/spnego.service.keytab</value>
</property>

<property>
  <name>hive.srv2.authentication.spnego.principal</name>
  <value>HTTP/_HOST@HADOOP.HPC.MUC</value>
</property>

<property>
  <name>hive.srv2.enable.doAs</name>
  <value>true</value>
</property>

<property>
  <name>hive.srv2.enable.impersonation</name>
  <value>true</value>
</property>

<property>
  <name>hive.srv2.logging.operation.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.srv2.logging.operation.log.location</name>
  <value>/etc/org.hadoop/tmp/hive/operation_logs</value>
</property>

<property>
  <name>hive.srv2.proxyuser.hue.groups</name>
  <value>*</value>
</property>

<property>
  <name>hive.srv2.proxyuser.hue.hosts</name>
  <value>srv3</value>
</property>

<property>
  <name>hive.srv2.support.dynamic.service.discovery</name>
  <value>true</value>
</property>

<property>
  <name>hive.srv2.table.type.mapping</name>
  <value>CLASSIC</value>
</property>

<property>
  <name>hive.srv2.tez.default.queues</name>
  <value>default</value>
</property>

<property>
  <name>hive.srv2.tez.initialize.default.sessions</name>
  <value>false</value>
</property>

<property>
  <name>hive.srv2.tez.sessions.per.default.queue</name>
  <value>1</value>
</property>

<property>
  <name>hive.srv2.thrift.http.path</name>
  <value>cliservice</value>
</property>

<property>
  <name>hive.srv2.thrift.http.port</name>
  <value>10001</value>
</property>

<property>
  <name>hive.srv2.thrift.max.worker.threads</name>
  <value>500</value>
</property>

<property>
  <name>hive.srv2.thrift.port</name>
  <value>10000</value>
</property>

<property>
  <name>hive.srv2.thrift.sasl.qop</name>
  <value>auth</value>
</property>

<property>
  <name>hive.srv2.transport.mode</name>
  <value>http</value>
</property>

<property>
  <name>hive.srv2.use.SSL</name>
  <value>false</value>
</property>

<property>
  <name>hive.srv2.zookeeper.namespace</name>
  <value>hivesrv2</value>
</property>

<property>
  <name>hive.smbjoin.cache.rows</name>
  <value>10000</value>
</property>

<property>
  <name>hive.stats.autogather</name>
  <value>true</value>
</property>

<property>
  <name>hive.stats.dbclass</name>
  <value>fs</value>
</property>

<property>
  <name>hive.stats.fetch.column.stats</name>
  <value>false</value>
</property>

<property>
  <name>hive.stats.fetch.partition.stats</name>
  <value>true</value>
</property>

<property>
  <name>hive.support.concurrency</name>
  <value>false</value>
</property>

<property>
  <name>hive.tez.auto.reducer.parallelism</name>
  <value>false</value>
</property>

<property>
  <name>hive.tez.container.size</name>
  <value>3584</value>
</property>

<property>
  <name>hive.tez.cpu.vcores</name>
  <value>-1</value>
</property>

<property>
  <name>hive.tez.dynamic.partition.pruning</name>
  <value>true</value>
</property>

<property>
  <name>hive.tez.dynamic.partition.pruning.max.data.size</name>
  <value>104857600</value>
</property>

<property>
  <name>hive.tez.dynamic.partition.pruning.max.event.size</name>
  <value>1048576</value>
</property>

<property>
  <name>hive.tez.input.format</name>
  <value>org.apache.hadoop.hive.ql.io.HiveInputFormat</value>
</property>

<property>
  <name>hive.tez.java.opts</name>
  <value>-server -Xmx2868m -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps</value>
</property>

<property>
  <name>hive.tez.log.level</name>
  <value>INFO</value>
</property>

<property>
  <name>hive.tez.max.partition.factor</name>
  <value>2.0</value>
</property>

<property>
  <name>hive.tez.min.partition.factor</name>
  <value>0.25</value>
</property>

<property>
  <name>hive.tez.smb.number.waves</name>
  <value>0.5</value>
</property>

<property>
  <name>hive.txn.manager</name>
  <value>org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager</value>
</property>

<property>
  <name>hive.txn.max.open.batch</name>
  <value>1000</value>
</property>

<property>
  <name>hive.txn.timeout</name>
  <value>300</value>
</property>

<property>
  <name>hive.user.install.directory</name>
  <value>/user/</value>
</property>

<property>
  <name>hive.vectorized.execution.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.tez.smb.number.waves</name>
  <value>0.5</value>
</property>

<property>
  <name>hive.txn.manager</name>
  <value>org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager</value>
</property>

<property>
  <name>hive.txn.max.open.batch</name>
  <value>1000</value>
</property>

<property>
  <name>hive.txn.timeout</name>
  <value>300</value>
</property>

<property>
  <name>hive.user.install.directory</name>
  <value>/user/</value>
</property>

<property>
  <name>hive.vectorized.execution.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.vectorized.execution.reduce.enabled</name>
  <value>false</value>
</property>

<property>
  <name>hive.vectorized.groupby.checkinterval</name>
  <value>4096</value>
</property>

<property>
  <name>hive.vectorized.groupby.flush.percent</name>
  <value>0.1</value>
</property>

<property>
  <name>hive.vectorized.groupby.maxentries</name>
  <value>100000</value>
</property>

<property>
  <name>hive.warehouse.subdir.inherit.perms</name>
  <value>true</value>
</property>

<property>
  <name>hive.zookeeper.client.port</name>
  <value>2181</value>
</property>

<property>
  <name>hive.zookeeper.namespace</name>
  <value>hive_zookeeper_namespace</value>
</property>

<property>
  <name>hive.zookeeper.quorum</name>
  <value>srv1:2181,srv2:2181,srv3:2181</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>org.postgresql.Driver</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:postgresql://srv3:5432/hive</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
</property>

0 个答案:

没有答案
相关问题
最新问题