UnionAll用于与spark scala中的列表具有不同列的数据帧

时间:2017-08-13 03:59:47

标签: scala apache-spark apache-spark-sql spark-dataframe user-defined-functions

我有一个数据帧列表:val dfs = Seq [df1,df2,df3,df4,....]

df1:

+--------+--------+
| Value1 | Amount |  
+--------+--------+
| A1     | 2000   |  
+--------+--------+
| B1     | 100    |  
+--------+--------+
| C1     | 349    |  
+--------+--------+

DF2

+--------+--------+
| value1 | Amount |
+--------+--------+
| A2     | 20     |
+--------+--------+
| B2     | 34     |
+--------+--------+
| C2     | 56     |
+--------+--------+

DF3:

+--------+--------+--------+
| value1 | value2 | amount |
+--------+--------+--------+
| A3     | CA     | 234    |
+--------+--------+--------+
| B3     | AZ     | 324    |
+--------+--------+--------+
| C3     | NY     | 213    |
+--------+--------+--------+

DF4:

+--------+--------+--------+
| value1 | value2 | amount |
+--------+--------+--------+
| A      | null   | 300    |
+--------+--------+--------+
| B      | null   | 400    |
+--------+--------+--------+
| C      | null   | 200    |
+--------+--------+--------+
| A1     | null   | 2000   |
+--------+--------+--------+
| B1     | null   | 100    |
+--------+--------+--------+
| C1     | null   | 349    |
+--------+--------+--------+
| A2     | null   | 20     |
+--------+--------+--------+
| B2     | null   | 34     |
+--------+--------+--------+
| C2     | null   | 56     |
+--------+--------+--------+
| A3     | CA     | 234    |
+--------+--------+--------+
| B3     | AZ     | 324    |
+--------+--------+--------+
| C3     | NY     | 213    |
+--------+--------+--------+

我希望将列表中的所有数据帧合并,但由于列数不同,无法执行此操作。 对于unionAll:当我做unionall时 dfs.reduce( unionall )。显示()

而不是unionall前三个数据帧,将第四列添加为null并使用df4执行unionall。有人可以帮助我如何做,unionall以通用的方式。从列表中获取数据帧

如何在循环条件下对具有不同列的数据帧列表进行unionAll操作?

预期产出:

/* Safari 7.1+ */
@media only screen and (max-device-width: 768px),
only screen and (-webkit-min-device-pixel-ratio:0) {
  .hero-section {
    sa-search-nav ion-row {
    background: none;
  }

  .search-form {
    ion-col.select {
      margin-top: 0px;
      margin-left: 0px;
    }
  }
}

  form.search-form {
    min-height: 300px;
  }
}

0 个答案:

没有答案