我正在尝试从describe formatted db.table
中检索“ Base Hive”分区的位置信息,并生成数据框。
我使用的是Spark 1.6,并且有一个Hive外部表,该表已分区在列dw_date
上,如果我运行hive Command describe formatted db.table
,则可以看到以下结果
+-------------------------------+----------------------------------------------------------------+-----------------------+--+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------------------+-----------------------+--+
| # col_name | data_type | comment |
| | NULL | NULL |
| id | string | |
| name | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| dw_date | string | |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | stage | NULL |
| Owner: | hive | NULL |
| CreateTime: | Thu May 30 15:11:06 EDT 2019 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Protect Mode: | None | NULL |
| Retention: | 0 | NULL |
| Location: | hdfs://devlocation1/share/myserver123/dev/stage/partition_chk | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | EXTERNAL | TRUE |
| | transient_lastDdlTime | 1559243466 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | field.delim | , |
| | line.delim | \n |
| | serialization.format | , |
+-------------------------------+----------------------------------------------------------------+-----------------------+--+
现在,从结果来看,我想要的只是将分区的基本位置保存到数据帧或rdd甚至是一个变量中(只是想以某种方式将其提取)。
我要从上述结果中检索的值是
hdfs://devlocation1/share/myserver123/dev/stage/partition_chk
我从以下方法入手:
df=sqlContext.sql("describe formatted stage.partition_chk")
df1=df.where(df[result].contains("Location:"))
但是,我无法从上方检索位置:信息。关于我们该如何做的任何建议?
谢谢, 席德