Question

此插入一张小桌子大约需要20-25秒。我有什么办法可以加快速度？我认为这可以在几秒钟内完成。我应该设置其他设置吗？

$ hive --hiveconf hive.execution.engine=mr --hiveconf hive.execution.engine=tez -e "insert into mytable select 'test', 'test', 'test', 'test', 'test', current_timestamp();"

使用

中的配置初始化日志记录

file:/etc/hive/conf.dist/hive-log4j2.properties Async: false
Query ID = xxxxx_20180925041644_dd645bd6-fd0c-4038-b668-21dca5481f67
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1533828187531_6078)

-----------------------------------------------------------------------------
VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
-----------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED      1          1        0        0       0       0
-----------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 4.88 s
-----------------------------------------------------------------------------
Loading data to table prodtracking.job_status_archive
-chgrp: 'domain users' does not match expected pattern for group
Usage: hadoop fs [generic options]
        [-appendToFile <localsrc> ... <dst>]
        [-cat [-ignoreCrc] <src> ...]
        [-checksum <src> ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
        [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] <path> ...]
        [-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
        [-createSnapshot <snapshotDir> [<snapshotName>]]
        [-deleteSnapshot <snapshotDir> <snapshotName>]
        [-df [-h] [<path> ...]]
        [-du [-s] [-h] [-x] <path> ...]
        [-expunge]
        [-find <path> ... <expression> ...]
        [-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-getfacl [-R] <path>]
        [-getfattr [-R] {-n name | -d} [-e en] <path>]
        [-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
        [-help [cmd ...]]
        [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
        [-mkdir [-p] <path> ...]
        [-moveFromLocal <localsrc> ... <dst>]
        [-moveToLocal <src> <localdst>]
        [-mv <src> ... <dst>]
        [-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
        [-renameSnapshot <snapshotDir> <oldName> <newName>]
        [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
        [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
        [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
        [-setfattr {-n name [-v value] | -x name} <path>]
        [-setrep [-R] [-w] <rep> <path> ...]
        [-stat [format] <path> ...]
        [-tail [-f] <file>]
        [-test -[defsz] <path>]
        [-text [-ignoreCrc] <src> ...]
        [-touchz <path> ...]
        [-truncate [-w] <length> <path> ...]
        [-usage [cmd ...]]

受支持的通用选项是 -conf指定应用程序配置文件 -D给定属性的使用值 -fs指定要使用的默认文件系统URL，从配置中覆盖'fs.defaultFS'属性。 -jt指定一个ResourceManager -files指定将逗号分隔的文件复制到map reduce集群 -libjars指定以逗号分隔的jar文件，以包含在类路径中。 -archives指定用逗号分隔的归档文件，这些归档文件将在计算机上取消归档。

一般命令行语法为命令[genericOptions] [commandOptions]

Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
OK
Time taken: 19.936 seconds

改进蜂巢中的单插入性能

0 个答案: