与Dataproc合作,我正在设置不同的配置以优化配置单元查询:
<property>
<name>hive.vectorized.execution.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.cbo.enable</name>
<value>true</value>
</property>
<property>
<name>hive.compute.query.using.stats</name>
<value>true</value>
</property>
<property>
<name>hive.stats.fetch.column.stats</name>
<value>true</value>
</property>
<property>
<name>hive.stats.fetch.partition.stats</name>
<value>true</value>
</property>
应用这些更改并重新启动hive meta-store和hiveserver后,由于不同节点上的连接问题,作业开始失败:
ERROR, exception: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException, result: %text org.apache.thrift.transport.TTransportException
但是,这些似乎是由于蜂巢mysql连接而发生的:
Oct 18 06:55:23 xxxx-spark-m /etc/mysql/debian-start[27552]: Upgrading MySQL tables if necessary.
Oct 18 06:55:23 xxxx-spark-m /etc/mysql/debian-start[27555]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Oct 18 06:55:23 xxxx-spark-m /etc/mysql/debian-start[27555]: Looking for 'mysql' as: /usr/bin/mysql
Oct 18 06:55:23 xxxx-spark-m /etc/mysql/debian-start[27555]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Oct 18 06:55:23 xxxx-spark-m /etc/mysql/debian-start[27555]: This installation of MySQL is already upgraded to 10.2.18-MariaDB, use --force if you still need to run mysql_upgrade
Oct 18 06:55:23 xxx-spark-m /etc/mysql/debian-start[27567]: Checking for insecure root accounts.
Oct 18 06:55:23 xxxx-spark-m /etc/mysql/debian-start[27571]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Oct 18 06:55:24 xxxx-spark-m mysqld[27488]: 2018-10-18 6:55:24 139690715899648 [Note] InnoDB: Buffer pool(s) load completed at 181018 6:55:24
Oct 18 07:05:46 xxxx-spark-m mysqld[27488]: 2018-10-18 7:05:46 139691201689344 [Warning] Aborted connection 42 to db: 'metastore' user: 'hive' host: 'host-xxx' (Got timeout reading communication packets)
我的问题首先显示为什么:
This installation of MySQL is already upgraded to 10.2.18-MariaDB, use --force if you still need to run mysql_upgrade
我也愿意访问mysql db来为hive用户创建特权,但是我无法弄清楚如何在dataproc上访问mysql db。