我对使用CTE创建的Hive(WITH子句)有一个看法,将两个表合并起来,然后进行计算以仅显示每个ID的最新记录。 在我的环境中,我有一个浏览配置单元数据库的工具(DBeaver,非datalake开发人员浏览数据所必需)。
查看代码
CREATE VIEW IF NOT EXISTS db.test_cte_view AS
with cte as (select * from db.test_cte union select * from db.test_cte_2),
tmp as (SELECT id, idate, ROW_NUMBER() over(PARTITION BY id ORDER BY idate desc ) AS row_num from cte)
SELECT cte.* from cte
join (SELECT * from tmp where tmp.row_num =1) tmp_2
on cte.id = tmp_2.id
and cte.idate = tmp_2.idate
问题是:
(这是我们在Hive中创建表和视图的主要方式)
我可以轻松地在DBeaver上浏览,但是,当运行spark进程以读取它时,它将失败,并显示以下内容:
##pyspark
spark.sql("select * from db.test_cte_view").show()
'Table or view not found: cte; line 3 pos 56'
Traceback (most recent call last):
File "DATA/fs3/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552132357519_15102/container_e378_1552132357519_15102_01_000001/pyspark.zip/pyspark/sql/session.py", line 545, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/DATA/fs3/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552132357519_15102/container_e378_1552132357519_15102_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/DATA/fs3/hadoop/yarn/local/usercache/ingouagn/appcache/application_1552132357519_15102/container_e378_1552132357519_15102_01_000001/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: 'Table or view not found: cte; line 3 pos 56'
我可以很好地阅读
##pyspark
spark.sql("select * from db.test_cte_view").show()
但是,当尝试使用DBeaver浏览时,它失败,并显示类似以下内容:
Query execution failed
Reason:
SQL Error [40000] [42000]: Error while compiling statement: FAILED: SemanticException line 1:330 Failed to recognize predicate 'UNION'. Failed rule: 'identifier' in subquery source in definition of VIEW test_cte_view [
SELECT `gen_attr_0` AS `id`, `gen_attr_1` AS `status`, `gen_attr_2` AS `idate` FROM (SELECT `gen_attr_0`, `gen_attr_1`, `gen_attr_2` FROM ((SELECT `gen_attr_0`, `gen_attr_1`, `gen_attr_2` FROM (SELECT `id` AS `gen_attr_0`, `status` AS `gen_attr_1`, `idate` AS `gen_attr_2` FROM `db`.`test_cte`) AS gen_subquery_0) UNION DISTINCT (SELECT `gen_attr_5`, `gen_attr_6`, `gen_attr_7` FROM (SELECT `id` AS `gen_attr_5`, `status` AS `gen_attr_6`, `idate` AS `gen_attr_7` FROM `db`.`test_cte_2`) AS gen_subquery_1)) AS cte INNER JOIN (SELECT `gen_attr_3`, `gen_attr_4`, `gen_attr_8` FROM (SELECT `gen_attr_3`, `gen_attr_4`, `gen_attr_8` FROM (SELECT gen_subquery_4.`gen_attr_3`, gen_subquery_4.`gen_attr_4`, row_number() OVER (PARTITION BY `gen_attr_3` ORDER BY `gen_attr_4` DESC NULLS LAST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS `gen_attr_8` FROM (SELECT `gen_attr_3`, `gen_attr_4` FROM ((SELECT `gen_attr_3`, `gen_attr_9`, `gen_attr_4` FROM (SELECT `id` AS `gen_attr_3`, `status` AS `gen_attr_9`, `idate` AS `gen_attr_4` FROM `db`.`test_cte`) AS gen_subquery_2) UNION DISTINCT (SELECT `gen_attr_5`, `gen_attr_6`, `gen_attr_7` FROM (SELECT `id` AS `gen_attr_5`, `status` AS `gen_attr_6`, `idate` AS `gen_attr_7` FROM `db`.`test_cte_2`) AS gen_subquery_3)) AS cte) AS gen_subquery_4) AS gen_subquery_5) AS tmp WHERE (`gen_attr_8` = 1)) AS tmp_2 ON ((`gen_attr_0` = `gen_attr_3`) AND (`gen_attr_2` = `gen_attr_4`))) AS cte
] used as test_cte_view at Line 1:14
在一种创建视图的方法与另一种创建方法之间,似乎生成的代码是不同的。
有没有办法使第一种情况(通过beeline创建视图并通过spark sql访问它)起作用?
谢谢。
火花:2.1.1 ,配置单元:1.2.1
CREATE TABLE db.test_cte(
id int,
status string,
idate date )
CREATE TABLE db.test_cte_2(
id int,
status string,
idate date )
填充:
insert into db.test_cte values
(1,"green","2019-03-08"),
(2,"green","2019-03-08"),
(3,"green","2019-03-08"),
(1,"red","2019-03-09"),
(1,"yellow","2019-03-10"),
(2,"gray","2019-03-09")
insert into db.test_cte_2 values
(10,"green","2019-03-08"),
(20,"green","2019-03-08"),
(30,"green","2019-03-08"),
(10,"red","2019-03-09"),
(10,"yellow","2019-03-10"),
(20,"gray","2019-03-09")
修改:
对于有兴趣的人,我在Spark JIRA上创建了一个问题:
https://issues.apache.org/jira/browse/SPARK-27203
答案 0 :(得分:0)
我在Spark2.1.1.2.6.1.0-129中遇到了相同的问题。升级到Spark2.4解决了该问题。
如果无法升级,则此变通办法在2.1上对我有用:
spark.table("db.my_view_with_ctes").registerTempTable("tmp")
spark.sql("select * from tmp")
这比在Spark2.4中通过spark-sql读取视图的运行时间长得多(对于我的用例而言,运行时间是其运行时间的10倍以上),但是它可以工作。