Question

这是我的现有数据框

+------------------+-------------------------+-----------+---------------+-------------------------+---------------------------+------------------------+--------------------------+---------------+-----------+----------------+-----------------+----------------------+--------------------------+-----------+--------------------+-----------+--------------------------------------------------------------------------------------------+-----------------------+------------------+-----------------------------+-----------------------+----------------------------------+
|DataPartition     |TimeStamp                |_lineItemId|_organizationId|fl:FinancialConceptGlobal|fl:FinancialConceptGlobalId|fl:FinancialConceptLocal|fl:FinancialConceptLocalId|fl:InstrumentId|fl:IsCredit|fl:IsDimensional|fl:IsRangeAllowed|fl:IsSegmentedByOrigin|fl:SegmentGroupDescription|fl:Segments|fl:StatementTypeCode|FFAction|!||LineItemName                                                                                |LineItemName.languageId|LocalLanguageLabel|LocalLanguageLabel.languageId|SegmentChildDescription|SegmentChildDescription.languageId|
+------------------+-------------------------+-----------+---------------+-------------------------+---------------------------+------------------------+--------------------------+---------------+-----------+----------------+-----------------+----------------------+--------------------------+-----------+--------------------+-----------+--------------------------------------------------------------------------------------------+-----------------------+------------------+-----------------------------+-----------------------+----------------------------------+
|SelfSourcedPrivate|2017-11-02T10:23:59+00:00|3          |4298009288     |XTOT                     |3016350                    |null                    |null                      |null           |true       |false           |false            |false                 |null                      |null       |BAL                 |I|!|       |Total Assets                                                                                |505074                 |null              |null                         |null                   |null                              |

这是上面数据框架的模式

root
 |-- DataPartition: string (nullable = true)
 |-- TimeStamp: string (nullable = true)
 |-- _lineItemId: long (nullable = true)
 |-- _organizationId: long (nullable = true)
 |-- fl:FinancialConceptGlobal: string (nullable = true)
 |-- fl:FinancialConceptGlobalId: long (nullable = true)
 |-- fl:FinancialConceptLocal: string (nullable = true)
 |-- fl:FinancialConceptLocalId: long (nullable = true)
 |-- fl:InstrumentId: long (nullable = true)
 |-- fl:IsCredit: boolean (nullable = true)
 |-- fl:IsDimensional: boolean (nullable = true)
 |-- fl:IsRangeAllowed: boolean (nullable = true)
 |-- fl:IsSegmentedByOrigin: boolean (nullable = true)
 |-- fl:SegmentGroupDescription: string (nullable = true)
 |-- fl:Segments: struct (nullable = true)
 |    |-- fl:SegmentSequence: struct (nullable = true)
 |    |    |-- _VALUE: long (nullable = true)
 |    |    |-- _segmentId: long (nullable = true)
 |-- fl:StatementTypeCode: string (nullable = true)
 |-- FFAction|!|: string (nullable = true)
 |-- LineItemName: string (nullable = true)
 |-- LineItemName.languageId: long (nullable = true)
 |-- LocalLanguageLabel: string (nullable = true)
 |-- LocalLanguageLabel.languageId: long (nullable = true)
 |-- SegmentChildDescription: string (nullable = true)
 |-- SegmentChildDescription.languageId: long (nullable = true)

我想使用下面的代码重命名数据框的标题列。

 val temp = dfTypeNew.select(dfTypeNew.columns.filter(x => !x.equals("fl:Segments")).map(x => col(x).as(x.replace("_", "LineItem_").replace("fl:", ""))): _*)

当我这样做时，我得到以下错误

线程中的异常＆＃34; main＆＃34; org.apache.spark.sql.AnalysisException：无法从LineItemName＃368中提取值：需要struct类型但得到了串;

当我在没有.的情况下重命名我的列时，我能够提取

Answer 1

错误存在，因为February, 2018 SELECT DATENAME(month, GETDATE()) AS 'Month Name'点用于访问(.)字段要读取具有列名称的字段，请使用下面的反引号

struct

输出

  val df = Seq(
    ("a","b","c"),
    ("a","b","c")
  ).toDF("x", "y", "z.z")

  df.select("x", "`z.z`").show(false)

希望这有帮助！

由Ramesh编辑

@Anupam，您所要做的就是使用Shankar在您的代码中建议的上述技术

+---+---+
|a  |c.c|
+---+---+
|a  |c  |
|a  |c  |
+---+---+

多数人。

从数据框架中提取值由于错误而导致错误。在spark中的列名称中

1 个答案: