在R中使用SparklyR更改嵌套的列名称

Question

我已经提到了这里提到的所有链接：

1）Link-1 2）Link-2 3）Link-3 4）Link-4

以下R代码已使用Sparklyr软件包编写。它读取巨大的JSON文件并创建数据库模式。

sc <- spark_connect(master = "local", config = conf, version = '2.2.0') # Connection
sample_tbl <- spark_read_json(sc,name="example",path="example.json", header = TRUE, 
                              memory = FALSE, overwrite = TRUE) # reads JSON file
sample_tbl <- sdf_schema_viewer(sample_tbl) # to create db schema
df <- tbl(sc,"example") # to create lookup table

它已创建以下数据库模式

现在

如果我重命名第一级列，那么它将起作用。

例如，

df %>% rename(ent = entities)

但是当我运行第二个深度嵌套列时，它不会重命名。

df %>% rename(e_hashtags = entities.hashtags)

显示错误：

Error in .f(.x[[i]], ...) : object 'entities.hashtags' not found

问题

我的问题是，如何也将第3到第4深层嵌套列重命名？

请参考上述数据库架构。

Answer 1

这样的火花不支持重命名单个嵌套字段。您必须铸造或重建整个结构。为了简单起见，我们假设数据如下所示：

cat('{"contributors": "foo", "coordinates": "bar", "entities": {"hashtags": ["foo", "bar"], "media": "missing"}}',  file = "/tmp/example.json")
df <- spark_read_json(sc, "df", "/tmp/example.json", overwrite=TRUE)

df %>% spark_dataframe() %>% invoke("schema") %>% invoke("treeString") %>% cat()

root
 |-- contributors: string (nullable = true)
 |-- coordinates: string (nullable = true)
 |-- entities: struct (nullable = true)
 |    |-- hashtags: array (nullable = true)
 |    |    |-- element: string (containsNull = true)
 |    |-- media: string (nullable = true)

具有简单的字符串表示形式：

df %>% 
  spark_dataframe() %>% 
  invoke("schema") %>% 
  invoke("simpleString") %>% 
  cat(sep = "\n")

struct<contributors:string,coordinates:string,entities:struct<hashtags:array<string>,media:string>>

使用强制转换时，您必须使用匹配的类型描述来定义表达式：

expr_cast <- invoke_static(
  sc, "org.apache.spark.sql.functions", "expr",
  "CAST(entities AS struct<e_hashtags:array<string>,media:string>)"
)

df_cast <- df %>% 
  spark_dataframe() %>% 
  invoke("withColumn", "entities", expr_cast) %>% 
  sdf_register()

df_cast %>% spark_dataframe() %>% invoke("schema") %>% invoke("treeString") %>% cat()

root
 |-- contributors: string (nullable = true)
 |-- coordinates: string (nullable = true)
 |-- entities: struct (nullable = true)
 |    |-- e_hashtags: array (nullable = true)
 |    |    |-- element: string (containsNull = true)
 |    |-- media: string (nullable = true)

要重建结构，您必须匹配所有组件：

expr_struct <- invoke_static(
  sc, "org.apache.spark.sql.functions", "expr",
  "struct(entities.hashtags AS e_hashtags, entities.media)"
)

df_struct <- df %>% 
  spark_dataframe() %>% 
  invoke("withColumn", "entities", expr_struct) %>% 
  sdf_register()

df_struct %>% spark_dataframe() %>% invoke("schema") %>% invoke("treeString") %>% cat()

root
 |-- contributors: string (nullable = true)
 |-- coordinates: string (nullable = true)
 |-- entities: struct (nullable = false)
 |    |-- e_hashtags: array (nullable = true)
 |    |    |-- element: string (containsNull = true)
 |    |-- media: string (nullable = true)